2023-10-17T00:28:42.435 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-17T00:28:42.436 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_rraja@teuthology --description rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/defaults msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} --verbose --name rraja-2023-10-17_00:10:32-rbd:cli-debug-62994-distro-default-smithi --archive /home/teuthworker/archive/rraja-2023-10-17_00:10:32-rbd:cli-debug-62994-distro-default-smithi/7430972 -- /home/teuthworker/archive/rraja-2023-10-17_00:10:32-rbd:cli-debug-62994-distro-default-smithi/7430972/orig.config.yaml 2023-10-17T00:28:42.466 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-17T00:28:42.541 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rraja-2023-10-17_00:10:32-rbd:cli-debug-62994-distro-default-smithi/7430972 branch: debug-62994 description: rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/defaults msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} email: rraja@redhat.com first_in_suite: false job_id: '7430972' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: rraja-2023-10-17_00:10:32-rbd:cli-debug-62994-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: debug-62994 ceph: conf: client: rbd default data pool: datapool rbd default features: 61 global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 debug rbd: 20 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: zlib bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: c3822f5f81e37dae2c0291e57dc72240858225fd ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: osd default pool size: 2 install: ceph: flavor: default sha1: c3822f5f81e37dae2c0291e57dc72240858225fd thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: debug-62994 sha1: c3822f5f81e37dae2c0291e57dc72240858225fd owner: scheduled_rraja@teuthology priority: 50 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 9882 sha1: c3822f5f81e37dae2c0291e57dc72240858225fd sleep_before_teardown: 0 suite: rbd:cli suite_branch: debug-62994 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_c3822f5f81e37dae2c0291e57dc72240858225fd/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: c3822f5f81e37dae2c0291e57dc72240858225fd targets: smithi169.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAaZyr1k69SkPlF/Xh4AZJUrxwBZlRxT2Tma5EXe7XCdHbtKN5MGae9jrpx30dbg96m2gGYzEUIuCckPOJylPVg= tasks: - install: null - ceph: null - exec: client.0: - sudo ceph osd pool create datapool 4 - rbd pool init datapool - install: extra_system_packages: - fio - workunit: clients: client.0: - rbd/rbd_support_module_recovery.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 8cdab074dcca9a68965bc5a50e9c30b691949723 timestamp: 2023-10-17_00:10:32 tube: smithi user: rraja verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-17T00:28:42.542 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_c3822f5f81e37dae2c0291e57dc72240858225fd/qa; will attempt to use it 2023-10-17T00:28:42.543 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_c3822f5f81e37dae2c0291e57dc72240858225fd/qa/tasks 2023-10-17T00:28:42.543 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-17T00:28:42.545 INFO:teuthology.task.internal:Checking packages... 2023-10-17T00:28:42.569 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'c3822f5f81e37dae2c0291e57dc72240858225fd' 2023-10-17T00:28:42.569 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-17T00:28:42.570 INFO:teuthology.packaging:ref: None 2023-10-17T00:28:42.570 INFO:teuthology.packaging:tag: None 2023-10-17T00:28:42.570 INFO:teuthology.packaging:branch: debug-62994 2023-10-17T00:28:42.571 INFO:teuthology.packaging:sha1: c3822f5f81e37dae2c0291e57dc72240858225fd 2023-10-17T00:28:42.571 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=debug-62994 2023-10-17T00:28:42.713 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6740-gc3822f5f-1jammy 2023-10-17T00:28:42.713 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-17T00:28:42.721 INFO:teuthology.task.internal:no buildpackages task found 2023-10-17T00:28:42.721 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-17T00:28:42.742 INFO:teuthology.task.internal:Saving configuration 2023-10-17T00:28:42.755 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-17T00:28:42.761 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-17T00:28:42.786 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi169.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rraja-2023-10-17_00:10:32-rbd:cli-debug-62994-distro-default-smithi/7430972', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-17 00:22:43.889635', 'locked_by': 'scheduled_rraja@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMyTsO2DRGa2T9N+753P0qrYpoaAMXMWtctwXi7k3Jor'} 2023-10-17T00:28:42.787 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-17T00:28:42.792 INFO:teuthology.task.internal:roles: ubuntu@smithi169.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-10-17T00:28:42.792 INFO:teuthology.run_tasks:Running task console_log... 2023-10-17T00:28:42.834 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7e8ead29d0>, signals=[15]) 2023-10-17T00:28:42.836 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-17T00:28:42.841 INFO:teuthology.task.internal:Opening connections... 2023-10-17T00:28:42.841 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi169.front.sepia.ceph.com 2023-10-17T00:28:42.846 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-17T00:28:42.924 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-17T00:28:42.930 DEBUG:teuthology.orchestra.run.smithi169:> uname -m 2023-10-17T00:28:43.035 INFO:teuthology.orchestra.run.smithi169.stdout:x86_64 2023-10-17T00:28:43.035 DEBUG:teuthology.orchestra.run.smithi169:> cat /etc/os-release 2023-10-17T00:28:43.080 INFO:teuthology.orchestra.run.smithi169.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-10-17T00:28:43.081 INFO:teuthology.orchestra.run.smithi169.stdout:NAME="Ubuntu" 2023-10-17T00:28:43.081 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION_ID="22.04" 2023-10-17T00:28:43.081 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-10-17T00:28:43.082 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION_CODENAME=jammy 2023-10-17T00:28:43.082 INFO:teuthology.orchestra.run.smithi169.stdout:ID=ubuntu 2023-10-17T00:28:43.082 INFO:teuthology.orchestra.run.smithi169.stdout:ID_LIKE=debian 2023-10-17T00:28:43.082 INFO:teuthology.orchestra.run.smithi169.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-17T00:28:43.082 INFO:teuthology.orchestra.run.smithi169.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-17T00:28:43.083 INFO:teuthology.orchestra.run.smithi169.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-17T00:28:43.083 INFO:teuthology.orchestra.run.smithi169.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-17T00:28:43.083 INFO:teuthology.orchestra.run.smithi169.stdout:UBUNTU_CODENAME=jammy 2023-10-17T00:28:43.084 INFO:teuthology.lock.ops:Updating smithi169.front.sepia.ceph.com on lock server 2023-10-17T00:28:43.104 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-17T00:28:43.112 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-17T00:28:43.117 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-17T00:28:43.118 DEBUG:teuthology.orchestra.run.smithi169:> test '!' -e /home/ubuntu/cephtest 2023-10-17T00:28:43.129 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-17T00:28:43.136 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-17T00:28:43.136 DEBUG:teuthology.orchestra.run.smithi169:> test -z $(ls -A /var/lib/ceph) 2023-10-17T00:28:43.194 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-17T00:28:43.229 INFO:teuthology.run_tasks:Running task kernel... 2023-10-17T00:28:43.243 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-17T00:28:43.244 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-17T00:28:43.245 DEBUG:teuthology.orchestra.run.smithi169:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-17T00:28:43.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-17T00:28:43.250 DEBUG:teuthology.orchestra.run.smithi169:> uname -r 2023-10-17T00:28:43.297 INFO:teuthology.orchestra.run.smithi169.stdout:5.15.0-56-generic 2023-10-17T00:28:43.298 INFO:teuthology.task.kernel:Running kernel on smithi169: 5.15.0-56-generic 2023-10-17T00:28:43.298 DEBUG:teuthology.orchestra.run.smithi169:> sudo apt-get clean 2023-10-17T00:28:43.437 DEBUG:teuthology.orchestra.run.smithi169:> sudo apt-get update 2023-10-17T00:28:43.580 INFO:teuthology.orchestra.run.smithi169.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-17T00:28:43.598 INFO:teuthology.orchestra.run.smithi169.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-10-17T00:28:43.599 INFO:teuthology.orchestra.run.smithi169.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-10-17T00:28:43.666 INFO:teuthology.orchestra.run.smithi169.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-10-17T00:28:43.810 INFO:teuthology.orchestra.run.smithi169.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [507 kB] 2023-10-17T00:28:43.877 INFO:teuthology.orchestra.run.smithi169.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,062 kB] 2023-10-17T00:28:43.894 INFO:teuthology.orchestra.run.smithi169.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [856 kB] 2023-10-17T00:28:43.901 INFO:teuthology.orchestra.run.smithi169.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [234 kB] 2023-10-17T00:28:43.905 INFO:teuthology.orchestra.run.smithi169.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.0 kB] 2023-10-17T00:28:43.906 INFO:teuthology.orchestra.run.smithi169.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.3 kB] 2023-10-17T00:28:43.906 INFO:teuthology.orchestra.run.smithi169.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [974 kB] 2023-10-17T00:28:43.920 INFO:teuthology.orchestra.run.smithi169.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [157 kB] 2023-10-17T00:28:43.922 INFO:teuthology.orchestra.run.smithi169.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [532 B] 2023-10-17T00:28:43.922 INFO:teuthology.orchestra.run.smithi169.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [992 kB] 2023-10-17T00:28:43.929 INFO:teuthology.orchestra.run.smithi169.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [658 kB] 2023-10-17T00:28:43.934 INFO:teuthology.orchestra.run.smithi169.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [217 kB] 2023-10-17T00:28:43.936 INFO:teuthology.orchestra.run.smithi169.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.0 kB] 2023-10-17T00:28:43.936 INFO:teuthology.orchestra.run.smithi169.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-10-17T00:28:43.937 INFO:teuthology.orchestra.run.smithi169.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-10-17T00:28:43.937 INFO:teuthology.orchestra.run.smithi169.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-10-17T00:28:43.953 INFO:teuthology.orchestra.run.smithi169.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-10-17T00:28:43.971 INFO:teuthology.orchestra.run.smithi169.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [41.7 kB] 2023-10-17T00:28:43.972 INFO:teuthology.orchestra.run.smithi169.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [33.9 kB] 2023-10-17T00:28:43.972 INFO:teuthology.orchestra.run.smithi169.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-10-17T00:28:43.972 INFO:teuthology.orchestra.run.smithi169.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-10-17T00:28:43.972 INFO:teuthology.orchestra.run.smithi169.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [13.4 kB] 2023-10-17T00:28:43.973 INFO:teuthology.orchestra.run.smithi169.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [24.3 kB] 2023-10-17T00:28:43.973 INFO:teuthology.orchestra.run.smithi169.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.4 kB] 2023-10-17T00:28:43.973 INFO:teuthology.orchestra.run.smithi169.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [640 B] 2023-10-17T00:28:44.009 INFO:teuthology.orchestra.run.smithi169.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [346 kB] 2023-10-17T00:28:44.015 INFO:teuthology.orchestra.run.smithi169.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [175 kB] 2023-10-17T00:28:44.021 INFO:teuthology.orchestra.run.smithi169.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2023-10-17T00:28:44.022 INFO:teuthology.orchestra.run.smithi169.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.0 kB] 2023-10-17T00:28:44.023 INFO:teuthology.orchestra.run.smithi169.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [953 kB] 2023-10-17T00:28:44.066 INFO:teuthology.orchestra.run.smithi169.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [154 kB] 2023-10-17T00:28:44.075 INFO:teuthology.orchestra.run.smithi169.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [532 B] 2023-10-17T00:28:44.075 INFO:teuthology.orchestra.run.smithi169.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [560 kB] 2023-10-17T00:28:44.090 INFO:teuthology.orchestra.run.smithi169.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [790 kB] 2023-10-17T00:28:44.112 INFO:teuthology.orchestra.run.smithi169.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [145 kB] 2023-10-17T00:28:44.115 INFO:teuthology.orchestra.run.smithi169.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2023-10-17T00:28:44.116 INFO:teuthology.orchestra.run.smithi169.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-10-17T00:28:44.117 INFO:teuthology.orchestra.run.smithi169.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-10-17T00:28:44.117 INFO:teuthology.orchestra.run.smithi169.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-10-17T00:28:44.138 INFO:teuthology.orchestra.run.smithi169.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-10-17T00:28:49.238 INFO:teuthology.orchestra.run.smithi169.stdout:Fetched 9,491 kB in 1s (6,897 kB/s) 2023-10-17T00:28:50.496 INFO:teuthology.orchestra.run.smithi169.stdout:Reading package lists... 2023-10-17T00:28:50.514 DEBUG:teuthology.orchestra.run.smithi169:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-17T00:28:50.565 INFO:teuthology.orchestra.run.smithi169.stdout:Reading package lists... 2023-10-17T00:28:50.729 INFO:teuthology.orchestra.run.smithi169.stdout:Building dependency tree... 2023-10-17T00:28:50.730 INFO:teuthology.orchestra.run.smithi169.stdout:Reading state information... 2023-10-17T00:28:50.868 INFO:teuthology.orchestra.run.smithi169.stdout:The following packages were automatically installed and are no longer required: 2023-10-17T00:28:50.869 INFO:teuthology.orchestra.run.smithi169.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-17T00:28:50.869 INFO:teuthology.orchestra.run.smithi169.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-17T00:28:50.869 INFO:teuthology.orchestra.run.smithi169.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-17T00:28:50.870 INFO:teuthology.orchestra.run.smithi169.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-10-17T00:28:50.870 INFO:teuthology.orchestra.run.smithi169.stdout: python2.7-minimal 2023-10-17T00:28:50.870 INFO:teuthology.orchestra.run.smithi169.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-17T00:28:50.871 INFO:teuthology.orchestra.run.smithi169.stdout:The following additional packages will be installed: 2023-10-17T00:28:50.871 INFO:teuthology.orchestra.run.smithi169.stdout: linux-generic linux-headers-5.15.0-86 linux-headers-5.15.0-86-generic 2023-10-17T00:28:50.871 INFO:teuthology.orchestra.run.smithi169.stdout: linux-headers-generic linux-image-5.15.0-86-generic 2023-10-17T00:28:50.871 INFO:teuthology.orchestra.run.smithi169.stdout: linux-modules-5.15.0-86-generic linux-modules-extra-5.15.0-86-generic 2023-10-17T00:28:50.872 INFO:teuthology.orchestra.run.smithi169.stdout:Suggested packages: 2023-10-17T00:28:50.872 INFO:teuthology.orchestra.run.smithi169.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-10-17T00:28:50.872 INFO:teuthology.orchestra.run.smithi169.stdout:Recommended packages: 2023-10-17T00:28:50.873 INFO:teuthology.orchestra.run.smithi169.stdout: thermald 2023-10-17T00:28:50.919 INFO:teuthology.orchestra.run.smithi169.stdout:The following NEW packages will be installed: 2023-10-17T00:28:50.919 INFO:teuthology.orchestra.run.smithi169.stdout: linux-headers-5.15.0-86 linux-headers-5.15.0-86-generic 2023-10-17T00:28:50.919 INFO:teuthology.orchestra.run.smithi169.stdout: linux-image-5.15.0-86-generic linux-modules-5.15.0-86-generic 2023-10-17T00:28:50.920 INFO:teuthology.orchestra.run.smithi169.stdout: linux-modules-extra-5.15.0-86-generic 2023-10-17T00:28:50.920 INFO:teuthology.orchestra.run.smithi169.stdout:The following packages will be upgraded: 2023-10-17T00:28:50.921 INFO:teuthology.orchestra.run.smithi169.stdout: linux-generic linux-headers-generic linux-image-generic 2023-10-17T00:28:50.983 INFO:teuthology.orchestra.run.smithi169.stdout:3 upgraded, 5 newly installed, 0 to remove and 268 not upgraded. 2023-10-17T00:28:50.983 INFO:teuthology.orchestra.run.smithi169.stdout:Need to get 113 MB of archives. 2023-10-17T00:28:50.984 INFO:teuthology.orchestra.run.smithi169.stdout:After this operation, 582 MB of additional disk space will be used. 2023-10-17T00:28:50.984 INFO:teuthology.orchestra.run.smithi169.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-86-generic amd64 5.15.0-86.96 [22.4 MB] 2023-10-17T00:28:51.694 INFO:teuthology.orchestra.run.smithi169.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-86-generic amd64 5.15.0-86.96 [11.5 MB] 2023-10-17T00:28:51.904 INFO:teuthology.orchestra.run.smithi169.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-86-generic amd64 5.15.0-86.96 [63.9 MB] 2023-10-17T00:28:53.308 INFO:teuthology.orchestra.run.smithi169.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.86.83 [1,702 B] 2023-10-17T00:28:53.308 INFO:teuthology.orchestra.run.smithi169.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.86.83 [2,472 B] 2023-10-17T00:28:53.308 INFO:teuthology.orchestra.run.smithi169.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-86 all 5.15.0-86.96 [12.3 MB] 2023-10-17T00:28:53.936 INFO:teuthology.orchestra.run.smithi169.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-86-generic amd64 5.15.0-86.96 [2,882 kB] 2023-10-17T00:28:54.092 INFO:teuthology.orchestra.run.smithi169.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.86.83 [2,328 B] 2023-10-17T00:28:54.513 INFO:teuthology.orchestra.run.smithi169.stdout:Fetched 113 MB in 3s (35.7 MB/s) 2023-10-17T00:28:54.671 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package linux-modules-5.15.0-86-generic. 2023-10-17T00:28:56.685 INFO:teuthology.orchestra.run.smithi169.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-10-17T00:28:56.689 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../0-linux-modules-5.15.0-86-generic_5.15.0-86.96_amd64.deb ... 2023-10-17T00:28:56.862 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking linux-modules-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-17T00:28:59.129 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package linux-image-5.15.0-86-generic. 2023-10-17T00:28:59.150 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../1-linux-image-5.15.0-86-generic_5.15.0-86.96_amd64.deb ... 2023-10-17T00:28:59.238 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking linux-image-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-17T00:29:00.099 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-86-generic. 2023-10-17T00:29:00.120 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-86-generic_5.15.0-86.96_amd64.deb ... 2023-10-17T00:29:00.149 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking linux-modules-extra-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-17T00:29:06.731 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../3-linux-generic_5.15.0.86.83_amd64.deb ... 2023-10-17T00:29:06.891 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking linux-generic (5.15.0.86.83) over (5.15.0.56.54) ... 2023-10-17T00:29:07.250 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.86.83_amd64.deb ... 2023-10-17T00:29:07.378 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking linux-image-generic (5.15.0.86.83) over (5.15.0.56.54) ... 2023-10-17T00:29:07.673 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package linux-headers-5.15.0-86. 2023-10-17T00:29:07.695 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../5-linux-headers-5.15.0-86_5.15.0-86.96_all.deb ... 2023-10-17T00:29:07.731 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking linux-headers-5.15.0-86 (5.15.0-86.96) ... 2023-10-17T00:29:13.724 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package linux-headers-5.15.0-86-generic. 2023-10-17T00:29:13.761 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../6-linux-headers-5.15.0-86-generic_5.15.0-86.96_amd64.deb ... 2023-10-17T00:29:13.799 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking linux-headers-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-17T00:29:17.097 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.86.83_amd64.deb ... 2023-10-17T00:29:17.228 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking linux-headers-generic (5.15.0.86.83) over (5.15.0.56.54) ... 2023-10-17T00:29:17.586 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up linux-headers-5.15.0-86 (5.15.0-86.96) ... 2023-10-17T00:29:17.708 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up linux-headers-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-17T00:29:17.834 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up linux-headers-generic (5.15.0.86.83) ... 2023-10-17T00:29:17.994 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up linux-image-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-17T00:29:19.402 INFO:teuthology.orchestra.run.smithi169.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-10-17T00:29:19.403 INFO:teuthology.orchestra.run.smithi169.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-10-17T00:29:19.403 INFO:teuthology.orchestra.run.smithi169.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-86-generic 2023-10-17T00:29:19.403 INFO:teuthology.orchestra.run.smithi169.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-86-generic 2023-10-17T00:29:19.691 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up linux-modules-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-17T00:29:21.116 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up linux-modules-extra-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-17T00:29:22.518 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up linux-image-generic (5.15.0.86.83) ... 2023-10-17T00:29:22.644 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up linux-generic (5.15.0.86.83) ... 2023-10-17T00:29:22.770 INFO:teuthology.orchestra.run.smithi169.stdout:Processing triggers for linux-image-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-17T00:29:22.845 INFO:teuthology.orchestra.run.smithi169.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-10-17T00:29:22.845 INFO:teuthology.orchestra.run.smithi169.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-86-generic 2023-10-17T00:29:37.116 INFO:teuthology.orchestra.run.smithi169.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-10-17T00:29:37.116 INFO:teuthology.orchestra.run.smithi169.stdout:Sourcing file `/etc/default/grub' 2023-10-17T00:29:37.142 INFO:teuthology.orchestra.run.smithi169.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-17T00:29:37.158 INFO:teuthology.orchestra.run.smithi169.stdout:Generating grub configuration file ... 2023-10-17T00:29:37.493 INFO:teuthology.orchestra.run.smithi169.stdout:Found linux image: /boot/vmlinuz-5.15.0-86-generic 2023-10-17T00:29:37.507 INFO:teuthology.orchestra.run.smithi169.stdout:Found initrd image: /boot/initrd.img-5.15.0-86-generic 2023-10-17T00:29:37.830 INFO:teuthology.orchestra.run.smithi169.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-17T00:29:37.836 INFO:teuthology.orchestra.run.smithi169.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-17T00:29:37.908 INFO:teuthology.orchestra.run.smithi169.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-17T00:29:37.913 INFO:teuthology.orchestra.run.smithi169.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-17T00:29:38.086 INFO:teuthology.orchestra.run.smithi169.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-17T00:29:38.086 INFO:teuthology.orchestra.run.smithi169.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-10-17T00:29:38.086 INFO:teuthology.orchestra.run.smithi169.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-17T00:29:38.135 INFO:teuthology.orchestra.run.smithi169.stdout:done 2023-10-17T00:29:38.646 DEBUG:teuthology.orchestra.run.smithi169:> dpkg -s linux-image-generic 2023-10-17T00:29:38.712 INFO:teuthology.orchestra.run.smithi169.stdout:Package: linux-image-generic 2023-10-17T00:29:38.712 INFO:teuthology.orchestra.run.smithi169.stdout:Status: install ok installed 2023-10-17T00:29:38.712 INFO:teuthology.orchestra.run.smithi169.stdout:Priority: optional 2023-10-17T00:29:38.713 INFO:teuthology.orchestra.run.smithi169.stdout:Section: kernel 2023-10-17T00:29:38.713 INFO:teuthology.orchestra.run.smithi169.stdout:Installed-Size: 20 2023-10-17T00:29:38.713 INFO:teuthology.orchestra.run.smithi169.stdout:Maintainer: Ubuntu Kernel Team 2023-10-17T00:29:38.713 INFO:teuthology.orchestra.run.smithi169.stdout:Architecture: amd64 2023-10-17T00:29:38.713 INFO:teuthology.orchestra.run.smithi169.stdout:Source: linux-meta 2023-10-17T00:29:38.714 INFO:teuthology.orchestra.run.smithi169.stdout:Version: 5.15.0.86.83 2023-10-17T00:29:38.714 INFO:teuthology.orchestra.run.smithi169.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-86), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-10-17T00:29:38.714 INFO:teuthology.orchestra.run.smithi169.stdout:Depends: linux-image-5.15.0-86-generic, linux-modules-extra-5.15.0-86-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-17T00:29:38.714 INFO:teuthology.orchestra.run.smithi169.stdout:Recommends: thermald 2023-10-17T00:29:38.714 INFO:teuthology.orchestra.run.smithi169.stdout:Description: Generic Linux kernel image 2023-10-17T00:29:38.715 INFO:teuthology.orchestra.run.smithi169.stdout: This package will always depend on the latest generic kernel image 2023-10-17T00:29:38.715 INFO:teuthology.orchestra.run.smithi169.stdout: available. 2023-10-17T00:29:38.716 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-86-generic 2023-10-17T00:29:38.716 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-17T00:29:38.716 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-17T00:29:38.717 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-17T00:29:38.717 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi169.front.sepia.ceph.com, path=None, version=distro) 2023-10-17T00:29:38.717 DEBUG:teuthology.orchestra.run.smithi169:> sudo apt-get clean 2023-10-17T00:29:38.902 DEBUG:teuthology.orchestra.run.smithi169:> sudo apt-get update 2023-10-17T00:29:39.019 INFO:teuthology.orchestra.run.smithi169.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-17T00:29:39.023 INFO:teuthology.orchestra.run.smithi169.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-17T00:29:39.028 INFO:teuthology.orchestra.run.smithi169.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-17T00:29:39.045 INFO:teuthology.orchestra.run.smithi169.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-17T00:29:40.288 INFO:teuthology.orchestra.run.smithi169.stdout:Reading package lists... 2023-10-17T00:29:40.306 DEBUG:teuthology.orchestra.run.smithi169:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-17T00:29:40.366 INFO:teuthology.orchestra.run.smithi169.stdout:Reading package lists... 2023-10-17T00:29:40.549 INFO:teuthology.orchestra.run.smithi169.stdout:Building dependency tree... 2023-10-17T00:29:40.550 INFO:teuthology.orchestra.run.smithi169.stdout:Reading state information... 2023-10-17T00:29:40.714 INFO:teuthology.orchestra.run.smithi169.stdout:linux-image-generic is already the newest version (5.15.0.86.83). 2023-10-17T00:29:40.714 INFO:teuthology.orchestra.run.smithi169.stdout:The following packages were automatically installed and are no longer required: 2023-10-17T00:29:40.715 INFO:teuthology.orchestra.run.smithi169.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-17T00:29:40.715 INFO:teuthology.orchestra.run.smithi169.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-17T00:29:40.715 INFO:teuthology.orchestra.run.smithi169.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-17T00:29:40.715 INFO:teuthology.orchestra.run.smithi169.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-17T00:29:40.716 INFO:teuthology.orchestra.run.smithi169.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-17T00:29:40.716 INFO:teuthology.orchestra.run.smithi169.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-17T00:29:40.751 INFO:teuthology.orchestra.run.smithi169.stdout:0 upgraded, 0 newly installed, 0 to remove and 268 not upgraded. 2023-10-17T00:29:40.753 DEBUG:teuthology.orchestra.run.smithi169:> dpkg -s linux-image-generic 2023-10-17T00:29:40.769 INFO:teuthology.orchestra.run.smithi169.stdout:Package: linux-image-generic 2023-10-17T00:29:40.769 INFO:teuthology.orchestra.run.smithi169.stdout:Status: install ok installed 2023-10-17T00:29:40.769 INFO:teuthology.orchestra.run.smithi169.stdout:Priority: optional 2023-10-17T00:29:40.770 INFO:teuthology.orchestra.run.smithi169.stdout:Section: kernel 2023-10-17T00:29:40.770 INFO:teuthology.orchestra.run.smithi169.stdout:Installed-Size: 20 2023-10-17T00:29:40.770 INFO:teuthology.orchestra.run.smithi169.stdout:Maintainer: Ubuntu Kernel Team 2023-10-17T00:29:40.770 INFO:teuthology.orchestra.run.smithi169.stdout:Architecture: amd64 2023-10-17T00:29:40.770 INFO:teuthology.orchestra.run.smithi169.stdout:Source: linux-meta 2023-10-17T00:29:40.771 INFO:teuthology.orchestra.run.smithi169.stdout:Version: 5.15.0.86.83 2023-10-17T00:29:40.771 INFO:teuthology.orchestra.run.smithi169.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-86), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-10-17T00:29:40.771 INFO:teuthology.orchestra.run.smithi169.stdout:Depends: linux-image-5.15.0-86-generic, linux-modules-extra-5.15.0-86-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-17T00:29:40.771 INFO:teuthology.orchestra.run.smithi169.stdout:Recommends: thermald 2023-10-17T00:29:40.771 INFO:teuthology.orchestra.run.smithi169.stdout:Description: Generic Linux kernel image 2023-10-17T00:29:40.772 INFO:teuthology.orchestra.run.smithi169.stdout: This package will always depend on the latest generic kernel image 2023-10-17T00:29:40.772 INFO:teuthology.orchestra.run.smithi169.stdout: available. 2023-10-17T00:29:40.772 DEBUG:teuthology.orchestra.run.smithi169:> mktemp 2023-10-17T00:29:40.816 INFO:teuthology.orchestra.run.smithi169.stdout:/tmp/tmp.Xr4VIfm6pa 2023-10-17T00:29:40.817 DEBUG:teuthology.orchestra.run.smithi169:> sudo cp /boot/grub/grub.cfg /tmp/tmp.Xr4VIfm6pa 2023-10-17T00:29:40.872 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0666 /tmp/tmp.Xr4VIfm6pa 2023-10-17T00:29:40.992 DEBUG:teuthology.orchestra.remote:smithi169:/tmp/tmp.Xr4VIfm6pa is 10KB 2023-10-17T00:29:41.006 DEBUG:teuthology.orchestra.run.smithi169:> rm -fr /tmp/tmp.Xr4VIfm6pa 2023-10-17T00:29:41.012 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-10-17T00:29:41.069 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-10-17T00:29:41.070 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-10-17T00:29:41.070 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-10-17T00:29:41.146 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-86-generic 2023-10-17T00:29:41.150 DEBUG:teuthology.orchestra.run.smithi169:> sudo update-grub 2023-10-17T00:29:42.408 INFO:teuthology.orchestra.run.smithi169.stderr:Sourcing file `/etc/default/grub' 2023-10-17T00:29:42.409 INFO:teuthology.orchestra.run.smithi169.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-17T00:29:42.415 INFO:teuthology.orchestra.run.smithi169.stderr:Generating grub configuration file ... 2023-10-17T00:29:42.591 INFO:teuthology.orchestra.run.smithi169.stderr:Found linux image: /boot/vmlinuz-5.15.0-86-generic 2023-10-17T00:29:42.597 INFO:teuthology.orchestra.run.smithi169.stderr:Found initrd image: /boot/initrd.img-5.15.0-86-generic 2023-10-17T00:29:42.853 INFO:teuthology.orchestra.run.smithi169.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-17T00:29:42.858 INFO:teuthology.orchestra.run.smithi169.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-17T00:29:42.900 INFO:teuthology.orchestra.run.smithi169.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-17T00:29:42.906 INFO:teuthology.orchestra.run.smithi169.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-17T00:29:43.066 INFO:teuthology.orchestra.run.smithi169.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-17T00:29:43.066 INFO:teuthology.orchestra.run.smithi169.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-10-17T00:29:43.066 INFO:teuthology.orchestra.run.smithi169.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-17T00:29:43.094 INFO:teuthology.orchestra.run.smithi169.stderr:done 2023-10-17T00:29:43.096 DEBUG:teuthology.orchestra.run.smithi169:> sudo shutdown -r now 2023-10-17T00:30:13.126 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-17T00:30:13.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-10-17T00:30:13.128 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-17T00:30:31.550 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-10-17T00:30:40.558 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-10-17T00:30:40.559 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-17T00:30:43.610 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-10-17T00:30:55.622 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-10-17T00:30:55.623 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-17T00:31:55.683 DEBUG:teuthology.orchestra.remote:timed out 2023-10-17T00:32:10.698 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-10-17T00:32:10.699 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-17T00:32:11.043 DEBUG:teuthology.orchestra.run.smithi169:> true 2023-10-17T00:32:11.694 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-10-17T00:32:11.694 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-86-generic"... 2023-10-17T00:32:11.694 DEBUG:teuthology.orchestra.run.smithi169:> uname -r 2023-10-17T00:32:11.741 INFO:teuthology.orchestra.run.smithi169.stdout:5.15.0-86-generic 2023-10-17T00:32:11.741 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-86-generic vs 5.15.0-86-generic 2023-10-17T00:32:11.741 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-17T00:32:11.742 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-10-17T00:32:12.743 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-17T00:32:12.743 DEBUG:teuthology.orchestra.run.smithi169:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-17T00:32:13.112 INFO:teuthology.orchestra.run.smithi169.stdout:ttyS1 2023-10-17T00:32:13.139 DEBUG:teuthology.parallel:result is None 2023-10-17T00:32:13.140 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-17T00:32:13.150 INFO:teuthology.task.internal:Creating test directory... 2023-10-17T00:32:13.150 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-17T00:32:13.156 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-17T00:32:13.162 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-17T00:32:13.169 INFO:teuthology.task.internal:Creating archive directory... 2023-10-17T00:32:13.170 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-17T00:32:13.222 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-17T00:32:13.229 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-17T00:32:13.229 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-17T00:32:13.282 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-17T00:32:13.290 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-17T00:32:13.292 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-17T00:32:13.299 INFO:teuthology.task.internal:Configuring sudo... 2023-10-17T00:32:13.299 DEBUG:teuthology.orchestra.run.smithi169:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-17T00:32:13.349 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-17T00:32:13.359 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-17T00:32:13.360 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-17T00:32:13.397 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-17T00:32:13.445 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-17T00:32:13.494 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-10-17T00:32:13.494 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-17T00:32:13.563 DEBUG:teuthology.orchestra.run.smithi169:> sudo service rsyslog restart 2023-10-17T00:32:13.696 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-17T00:32:13.704 INFO:teuthology.task.internal:Starting timer... 2023-10-17T00:32:13.705 INFO:teuthology.run_tasks:Running task pcp... 2023-10-17T00:32:13.717 INFO:teuthology.run_tasks:Running task selinux... 2023-10-17T00:32:13.725 DEBUG:teuthology.task.selinux:Excluding smithi169: OS 'ubuntu' does not support SELinux 2023-10-17T00:32:13.726 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-17T00:32:13.726 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-10-17T00:32:13.727 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-17T00:32:13.727 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-17T00:32:13.736 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-17T00:32:13.742 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-17T00:32:13.881 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-17T00:32:13.896 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-17T00:32:13.898 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi169.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-17T00:39:53.207 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi169.front.sepia.ceph.com')] 2023-10-17T00:39:53.209 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-10-17T00:39:53.210 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-17T00:39:53.290 DEBUG:teuthology.orchestra.run.smithi169:> true 2023-10-17T00:39:53.407 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-10-17T00:39:53.407 INFO:teuthology.run_tasks:Running task clock... 2023-10-17T00:39:53.418 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-17T00:39:53.419 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-17T00:39:53.420 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-17T00:39:53.477 INFO:teuthology.orchestra.run.smithi169.stdout:17 Oct 00:39:53 ntpd[27992]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-10-17T00:39:53.477 INFO:teuthology.orchestra.run.smithi169.stdout:17 Oct 00:39:53 ntpd[27992]: Command line: ntpd -gq 2023-10-17T00:39:53.478 INFO:teuthology.orchestra.run.smithi169.stdout:17 Oct 00:39:53 ntpd[27992]: ---------------------------------------------------- 2023-10-17T00:39:53.478 INFO:teuthology.orchestra.run.smithi169.stdout:17 Oct 00:39:53 ntpd[27992]: ntp-4 is maintained by Network Time Foundation, 2023-10-17T00:39:53.478 INFO:teuthology.orchestra.run.smithi169.stdout:17 Oct 00:39:53 ntpd[27992]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-10-17T00:39:53.478 INFO:teuthology.orchestra.run.smithi169.stdout:17 Oct 00:39:53 ntpd[27992]: corporation. Support and training for ntp-4 are 2023-10-17T00:39:53.479 INFO:teuthology.orchestra.run.smithi169.stdout:17 Oct 00:39:53 ntpd[27992]: available at https://www.nwtime.org/support 2023-10-17T00:39:53.479 INFO:teuthology.orchestra.run.smithi169.stdout:17 Oct 00:39:53 ntpd[27992]: ---------------------------------------------------- 2023-10-17T00:39:53.479 INFO:teuthology.orchestra.run.smithi169.stdout:17 Oct 00:39:53 ntpd[27992]: proto: precision = 0.046 usec (-24) 2023-10-17T00:39:53.480 INFO:teuthology.orchestra.run.smithi169.stdout:17 Oct 00:39:53 ntpd[27992]: basedate set to 2022-02-04 2023-10-17T00:39:53.480 INFO:teuthology.orchestra.run.smithi169.stdout:17 Oct 00:39:53 ntpd[27992]: gps base set to 2022-02-06 (week 2196) 2023-10-17T00:39:53.480 INFO:teuthology.orchestra.run.smithi169.stdout:17 Oct 00:39:53 ntpd[27992]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-17T00:39:53.480 INFO:teuthology.orchestra.run.smithi169.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-17T00:39:53.481 INFO:teuthology.orchestra.run.smithi169.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-17T00:39:53.482 INFO:teuthology.orchestra.run.smithi169.stdout:17 Oct 00:39:53 ntpd[27992]: restrict ::: KOD does nothing without LIMITED. 2023-10-17T00:39:53.482 INFO:teuthology.orchestra.run.smithi169.stdout:17 Oct 00:39:53 ntpd[27992]: Listen and drop on 0 v6wildcard [::]:123 2023-10-17T00:39:53.482 INFO:teuthology.orchestra.run.smithi169.stdout:17 Oct 00:39:53 ntpd[27992]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-17T00:39:53.483 INFO:teuthology.orchestra.run.smithi169.stdout:17 Oct 00:39:53 ntpd[27992]: Listen normally on 2 lo 127.0.0.1:123 2023-10-17T00:39:53.483 INFO:teuthology.orchestra.run.smithi169.stdout:17 Oct 00:39:53 ntpd[27992]: Listen normally on 3 enp3s0f1 172.21.15.169:123 2023-10-17T00:39:53.483 INFO:teuthology.orchestra.run.smithi169.stdout:17 Oct 00:39:53 ntpd[27992]: Listen normally on 4 lo [::1]:123 2023-10-17T00:39:53.483 INFO:teuthology.orchestra.run.smithi169.stdout:17 Oct 00:39:53 ntpd[27992]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7f4d%5]:123 2023-10-17T00:39:53.483 INFO:teuthology.orchestra.run.smithi169.stdout:17 Oct 00:39:53 ntpd[27992]: Listening on routing socket on fd #22 for interface updates 2023-10-17T00:39:54.478 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:39:54 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-17T00:39:54.478 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:39:54 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231017: Permission denied 2023-10-17T00:39:54.479 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:39:54 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-17T00:39:54.479 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:39:54 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231017: Permission denied 2023-10-17T00:39:54.479 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:39:54 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-17T00:39:54.479 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:39:54 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231017: Permission denied 2023-10-17T00:39:54.479 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:39:54 ntpd[27992]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-17T00:39:54.480 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:39:54 ntpd[27992]: can't open /var/log/ntpstats/peerstats.20231017: Permission denied 2023-10-17T00:39:56.478 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:39:56 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-17T00:39:56.478 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:39:56 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231017: Permission denied 2023-10-17T00:39:56.478 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:39:56 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-17T00:39:56.479 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:39:56 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231017: Permission denied 2023-10-17T00:39:56.479 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:39:56 ntpd[27992]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-17T00:39:56.479 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:39:56 ntpd[27992]: can't open /var/log/ntpstats/peerstats.20231017: Permission denied 2023-10-17T00:39:58.479 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:39:58 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-17T00:39:58.479 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:39:58 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231017: Permission denied 2023-10-17T00:39:58.480 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:39:58 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-17T00:39:58.480 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:39:58 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231017: Permission denied 2023-10-17T00:39:58.480 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:39:58 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-17T00:39:58.480 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:39:58 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231017: Permission denied 2023-10-17T00:39:58.481 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:39:58 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-17T00:39:58.481 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:39:58 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231017: Permission denied 2023-10-17T00:39:58.481 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:39:58 ntpd[27992]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-17T00:39:58.481 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:39:58 ntpd[27992]: can't open /var/log/ntpstats/peerstats.20231017: Permission denied 2023-10-17T00:39:58.481 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:39:58 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-17T00:39:58.481 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:39:58 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231017: Permission denied 2023-10-17T00:39:58.482 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:39:58 ntpd[27992]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-17T00:39:58.482 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:39:58 ntpd[27992]: can't open /var/log/ntpstats/peerstats.20231017: Permission denied 2023-10-17T00:39:58.482 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:39:58 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-17T00:39:58.482 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:39:58 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231017: Permission denied 2023-10-17T00:39:58.483 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:39:58 ntpd[27992]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-17T00:39:58.483 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:39:58 ntpd[27992]: can't open /var/log/ntpstats/peerstats.20231017: Permission denied 2023-10-17T00:40:00.479 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:40:00 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-17T00:40:00.479 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:40:00 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231017: Permission denied 2023-10-17T00:40:00.479 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:40:00 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-17T00:40:00.480 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:40:00 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231017: Permission denied 2023-10-17T00:40:00.480 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:40:00 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-17T00:40:00.480 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:40:00 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231017: Permission denied 2023-10-17T00:40:00.480 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:40:00 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-17T00:40:00.480 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:40:00 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231017: Permission denied 2023-10-17T00:40:00.480 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:40:00 ntpd[27992]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-17T00:40:00.481 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:40:00 ntpd[27992]: can't open /var/log/ntpstats/peerstats.20231017: Permission denied 2023-10-17T00:40:00.481 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:40:00 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-17T00:40:00.481 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:40:00 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231017: Permission denied 2023-10-17T00:40:00.481 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:40:00 ntpd[27992]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-17T00:40:00.481 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:40:00 ntpd[27992]: can't open /var/log/ntpstats/peerstats.20231017: Permission denied 2023-10-17T00:40:00.482 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:40:00 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-17T00:40:00.482 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:40:00 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231017: Permission denied 2023-10-17T00:40:00.482 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:40:00 ntpd[27992]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-17T00:40:00.482 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:40:00 ntpd[27992]: can't open /var/log/ntpstats/peerstats.20231017: Permission denied 2023-10-17T00:40:00.483 INFO:teuthology.orchestra.run.smithi169.stdout:17 Oct 00:40:00 ntpd[27992]: ntpd: time slew -0.001185 s 2023-10-17T00:40:00.483 INFO:teuthology.orchestra.run.smithi169.stdout:ntpd: time slew -0.001185s 2023-10-17T00:40:00.484 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:40:00 ntpd[27992]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-17T00:40:00.484 INFO:teuthology.orchestra.run.smithi169.stderr:17 Oct 00:40:00 ntpd[27992]: can't open /var/log/ntpstats/loopstats.20231017: Permission denied 2023-10-17T00:40:00.571 INFO:teuthology.orchestra.run.smithi169.stdout: remote refid st t when poll reach delay offset jitter 2023-10-17T00:40:00.572 INFO:teuthology.orchestra.run.smithi169.stdout:============================================================================== 2023-10-17T00:40:00.572 INFO:teuthology.orchestra.run.smithi169.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-17T00:40:00.573 INFO:teuthology.orchestra.run.smithi169.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-17T00:40:00.573 INFO:teuthology.orchestra.run.smithi169.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-17T00:40:00.573 INFO:teuthology.orchestra.run.smithi169.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-17T00:40:00.574 INFO:teuthology.run_tasks:Running task install... 2023-10-17T00:40:00.582 DEBUG:teuthology.task.install:project ceph 2023-10-17T00:40:00.583 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c3822f5f81e37dae2c0291e57dc72240858225fd'}} 2023-10-17T00:40:00.583 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'c3822f5f81e37dae2c0291e57dc72240858225fd'} 2023-10-17T00:40:00.583 INFO:teuthology.task.install:Using flavor: default 2023-10-17T00:40:00.589 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-17T00:40:00.590 INFO:teuthology.task.install:extra packages: [] 2023-10-17T00:40:00.591 DEBUG:teuthology.orchestra.run.smithi169:> sudo apt-key list | grep Ceph 2023-10-17T00:40:00.675 INFO:teuthology.orchestra.run.smithi169.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-17T00:40:00.701 INFO:teuthology.orchestra.run.smithi169.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-17T00:40:00.701 INFO:teuthology.orchestra.run.smithi169.stdout:uid [ unknown] Ceph.com (release key) 2023-10-17T00:40:00.702 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-10-17T00:40:00.703 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=c3822f5f81e37dae2c0291e57dc72240858225fd 2023-10-17T00:40:00.856 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default/ 2023-10-17T00:40:00.857 INFO:teuthology.task.install.deb:Package version is 18.0.0-6740-gc3822f5f-1jammy 2023-10-17T00:40:00.961 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-10-17T00:40:00.961 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-17T00:40:00.975 DEBUG:teuthology.orchestra.run.smithi169:> sudo apt-get update 2023-10-17T00:40:01.161 INFO:teuthology.orchestra.run.smithi169.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-17T00:40:01.161 INFO:teuthology.orchestra.run.smithi169.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-17T00:40:01.170 INFO:teuthology.orchestra.run.smithi169.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-17T00:40:01.187 INFO:teuthology.orchestra.run.smithi169.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-17T00:40:01.254 INFO:teuthology.orchestra.run.smithi169.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy InRelease 2023-10-17T00:40:01.291 INFO:teuthology.orchestra.run.smithi169.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-10-17T00:40:01.327 INFO:teuthology.orchestra.run.smithi169.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-17T00:40:01.364 INFO:teuthology.orchestra.run.smithi169.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,422 B] 2023-10-17T00:40:01.408 INFO:teuthology.orchestra.run.smithi169.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-10-17T00:40:01.503 INFO:teuthology.orchestra.run.smithi169.stdout:Fetched 28.6 kB in 0s (81.2 kB/s) 2023-10-17T00:40:02.349 INFO:teuthology.orchestra.run.smithi169.stdout:Reading package lists... 2023-10-17T00:40:02.365 DEBUG:teuthology.orchestra.run.smithi169:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6740-gc3822f5f-1jammy cephadm=18.0.0-6740-gc3822f5f-1jammy ceph-mds=18.0.0-6740-gc3822f5f-1jammy ceph-mgr=18.0.0-6740-gc3822f5f-1jammy ceph-common=18.0.0-6740-gc3822f5f-1jammy ceph-fuse=18.0.0-6740-gc3822f5f-1jammy ceph-test=18.0.0-6740-gc3822f5f-1jammy ceph-volume=18.0.0-6740-gc3822f5f-1jammy radosgw=18.0.0-6740-gc3822f5f-1jammy python3-rados=18.0.0-6740-gc3822f5f-1jammy python3-rgw=18.0.0-6740-gc3822f5f-1jammy python3-cephfs=18.0.0-6740-gc3822f5f-1jammy python3-rbd=18.0.0-6740-gc3822f5f-1jammy libcephfs2=18.0.0-6740-gc3822f5f-1jammy libcephfs-dev=18.0.0-6740-gc3822f5f-1jammy librados2=18.0.0-6740-gc3822f5f-1jammy librbd1=18.0.0-6740-gc3822f5f-1jammy rbd-fuse=18.0.0-6740-gc3822f5f-1jammy 2023-10-17T00:40:02.427 INFO:teuthology.orchestra.run.smithi169.stdout:Reading package lists... 2023-10-17T00:40:02.611 INFO:teuthology.orchestra.run.smithi169.stdout:Building dependency tree... 2023-10-17T00:40:02.611 INFO:teuthology.orchestra.run.smithi169.stdout:Reading state information... 2023-10-17T00:40:02.777 INFO:teuthology.orchestra.run.smithi169.stdout:The following packages were automatically installed and are no longer required: 2023-10-17T00:40:02.778 INFO:teuthology.orchestra.run.smithi169.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-17T00:40:02.778 INFO:teuthology.orchestra.run.smithi169.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-17T00:40:02.778 INFO:teuthology.orchestra.run.smithi169.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-17T00:40:02.778 INFO:teuthology.orchestra.run.smithi169.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-17T00:40:02.780 INFO:teuthology.orchestra.run.smithi169.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-17T00:40:02.780 INFO:teuthology.orchestra.run.smithi169.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-17T00:40:02.781 INFO:teuthology.orchestra.run.smithi169.stdout:The following additional packages will be installed: 2023-10-17T00:40:02.781 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-17T00:40:02.781 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-10-17T00:40:02.781 INFO:teuthology.orchestra.run.smithi169.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-17T00:40:02.782 INFO:teuthology.orchestra.run.smithi169.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-10-17T00:40:02.782 INFO:teuthology.orchestra.run.smithi169.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-10-17T00:40:02.782 INFO:teuthology.orchestra.run.smithi169.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-10-17T00:40:02.782 INFO:teuthology.orchestra.run.smithi169.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-10-17T00:40:02.782 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-10-17T00:40:02.783 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-10-17T00:40:02.783 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-10-17T00:40:02.783 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-10-17T00:40:02.783 INFO:teuthology.orchestra.run.smithi169.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-10-17T00:40:02.783 INFO:teuthology.orchestra.run.smithi169.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-10-17T00:40:02.784 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-10-17T00:40:02.784 INFO:teuthology.orchestra.run.smithi169.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-10-17T00:40:02.784 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2023-10-17T00:40:02.784 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2023-10-17T00:40:02.784 INFO:teuthology.orchestra.run.smithi169.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-17T00:40:02.785 INFO:teuthology.orchestra.run.smithi169.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-17T00:40:02.785 INFO:teuthology.orchestra.run.smithi169.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-17T00:40:02.785 INFO:teuthology.orchestra.run.smithi169.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-17T00:40:02.785 INFO:teuthology.orchestra.run.smithi169.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2023-10-17T00:40:02.786 INFO:teuthology.orchestra.run.smithi169.stdout:Suggested packages: 2023-10-17T00:40:02.786 INFO:teuthology.orchestra.run.smithi169.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-10-17T00:40:02.786 INFO:teuthology.orchestra.run.smithi169.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-10-17T00:40:02.786 INFO:teuthology.orchestra.run.smithi169.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-10-17T00:40:02.786 INFO:teuthology.orchestra.run.smithi169.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-10-17T00:40:02.787 INFO:teuthology.orchestra.run.smithi169.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-10-17T00:40:02.787 INFO:teuthology.orchestra.run.smithi169.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-10-17T00:40:02.787 INFO:teuthology.orchestra.run.smithi169.stdout: python3-watchdog 2023-10-17T00:40:02.787 INFO:teuthology.orchestra.run.smithi169.stdout:Recommended packages: 2023-10-17T00:40:02.787 INFO:teuthology.orchestra.run.smithi169.stdout: btrfs-tools 2023-10-17T00:40:02.880 INFO:teuthology.orchestra.run.smithi169.stdout:The following NEW packages will be installed: 2023-10-17T00:40:02.881 INFO:teuthology.orchestra.run.smithi169.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-17T00:40:02.881 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-17T00:40:02.881 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2023-10-17T00:40:02.882 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2023-10-17T00:40:02.882 INFO:teuthology.orchestra.run.smithi169.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2023-10-17T00:40:02.882 INFO:teuthology.orchestra.run.smithi169.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-10-17T00:40:02.883 INFO:teuthology.orchestra.run.smithi169.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-10-17T00:40:02.883 INFO:teuthology.orchestra.run.smithi169.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-10-17T00:40:02.883 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-10-17T00:40:02.884 INFO:teuthology.orchestra.run.smithi169.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2023-10-17T00:40:02.884 INFO:teuthology.orchestra.run.smithi169.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-10-17T00:40:02.884 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-10-17T00:40:02.884 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-10-17T00:40:02.884 INFO:teuthology.orchestra.run.smithi169.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-10-17T00:40:02.885 INFO:teuthology.orchestra.run.smithi169.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-10-17T00:40:02.885 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-10-17T00:40:02.885 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-10-17T00:40:02.885 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2023-10-17T00:40:02.885 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-10-17T00:40:02.886 INFO:teuthology.orchestra.run.smithi169.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-17T00:40:02.886 INFO:teuthology.orchestra.run.smithi169.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-17T00:40:02.886 INFO:teuthology.orchestra.run.smithi169.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-17T00:40:02.886 INFO:teuthology.orchestra.run.smithi169.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-17T00:40:02.886 INFO:teuthology.orchestra.run.smithi169.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-10-17T00:40:02.887 INFO:teuthology.orchestra.run.smithi169.stdout: socat xmlstarlet 2023-10-17T00:40:02.938 INFO:teuthology.orchestra.run.smithi169.stdout:0 upgraded, 109 newly installed, 0 to remove and 256 not upgraded. 2023-10-17T00:40:02.939 INFO:teuthology.orchestra.run.smithi169.stdout:Need to get 184 MB of archives. 2023-10-17T00:40:02.939 INFO:teuthology.orchestra.run.smithi169.stdout:After this operation, 824 MB of additional disk space will be used. 2023-10-17T00:40:02.939 INFO:teuthology.orchestra.run.smithi169.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-10-17T00:40:02.996 INFO:teuthology.orchestra.run.smithi169.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-10-17T00:40:03.031 INFO:teuthology.orchestra.run.smithi169.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-10-17T00:40:03.034 INFO:teuthology.orchestra.run.smithi169.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-10-17T00:40:03.049 INFO:teuthology.orchestra.run.smithi169.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-10-17T00:40:03.102 INFO:teuthology.orchestra.run.smithi169.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-10-17T00:40:03.107 INFO:teuthology.orchestra.run.smithi169.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-10-17T00:40:03.109 INFO:teuthology.orchestra.run.smithi169.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6740-gc3822f5f-1jammy [3,450 kB] 2023-10-17T00:40:03.117 INFO:teuthology.orchestra.run.smithi169.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-10-17T00:40:03.121 INFO:teuthology.orchestra.run.smithi169.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-10-17T00:40:03.122 INFO:teuthology.orchestra.run.smithi169.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-10-17T00:40:03.122 INFO:teuthology.orchestra.run.smithi169.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-10-17T00:40:03.130 INFO:teuthology.orchestra.run.smithi169.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-10-17T00:40:03.131 INFO:teuthology.orchestra.run.smithi169.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-10-17T00:40:03.132 INFO:teuthology.orchestra.run.smithi169.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-10-17T00:40:03.132 INFO:teuthology.orchestra.run.smithi169.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-10-17T00:40:03.140 INFO:teuthology.orchestra.run.smithi169.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-10-17T00:40:03.141 INFO:teuthology.orchestra.run.smithi169.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-10-17T00:40:03.141 INFO:teuthology.orchestra.run.smithi169.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-10-17T00:40:03.157 INFO:teuthology.orchestra.run.smithi169.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-10-17T00:40:03.157 INFO:teuthology.orchestra.run.smithi169.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-10-17T00:40:03.158 INFO:teuthology.orchestra.run.smithi169.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-10-17T00:40:03.160 INFO:teuthology.orchestra.run.smithi169.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-10-17T00:40:03.162 INFO:teuthology.orchestra.run.smithi169.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-10-17T00:40:03.162 INFO:teuthology.orchestra.run.smithi169.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-10-17T00:40:03.163 INFO:teuthology.orchestra.run.smithi169.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-10-17T00:40:03.173 INFO:teuthology.orchestra.run.smithi169.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-17T00:40:03.174 INFO:teuthology.orchestra.run.smithi169.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-10-17T00:40:03.174 INFO:teuthology.orchestra.run.smithi169.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-10-17T00:40:03.190 INFO:teuthology.orchestra.run.smithi169.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-10-17T00:40:03.191 INFO:teuthology.orchestra.run.smithi169.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-10-17T00:40:03.192 INFO:teuthology.orchestra.run.smithi169.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-10-17T00:40:03.197 INFO:teuthology.orchestra.run.smithi169.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-10-17T00:40:03.197 INFO:teuthology.orchestra.run.smithi169.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-10-17T00:40:03.198 INFO:teuthology.orchestra.run.smithi169.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-10-17T00:40:03.198 INFO:teuthology.orchestra.run.smithi169.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-10-17T00:40:03.207 INFO:teuthology.orchestra.run.smithi169.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-10-17T00:40:03.209 INFO:teuthology.orchestra.run.smithi169.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-10-17T00:40:03.210 INFO:teuthology.orchestra.run.smithi169.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-10-17T00:40:03.224 INFO:teuthology.orchestra.run.smithi169.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-10-17T00:40:03.225 INFO:teuthology.orchestra.run.smithi169.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-10-17T00:40:03.228 INFO:teuthology.orchestra.run.smithi169.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-10-17T00:40:03.229 INFO:teuthology.orchestra.run.smithi169.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-10-17T00:40:03.232 INFO:teuthology.orchestra.run.smithi169.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-10-17T00:40:03.285 INFO:teuthology.orchestra.run.smithi169.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-10-17T00:40:03.286 INFO:teuthology.orchestra.run.smithi169.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-10-17T00:40:03.286 INFO:teuthology.orchestra.run.smithi169.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-10-17T00:40:03.287 INFO:teuthology.orchestra.run.smithi169.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-17T00:40:03.287 INFO:teuthology.orchestra.run.smithi169.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-10-17T00:40:03.288 INFO:teuthology.orchestra.run.smithi169.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-10-17T00:40:03.306 INFO:teuthology.orchestra.run.smithi169.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-10-17T00:40:03.308 INFO:teuthology.orchestra.run.smithi169.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-10-17T00:40:03.308 INFO:teuthology.orchestra.run.smithi169.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-10-17T00:40:03.323 INFO:teuthology.orchestra.run.smithi169.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-10-17T00:40:03.323 INFO:teuthology.orchestra.run.smithi169.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-10-17T00:40:03.324 INFO:teuthology.orchestra.run.smithi169.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-10-17T00:40:03.324 INFO:teuthology.orchestra.run.smithi169.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-10-17T00:40:03.324 INFO:teuthology.orchestra.run.smithi169.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-10-17T00:40:03.325 INFO:teuthology.orchestra.run.smithi169.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-10-17T00:40:03.336 INFO:teuthology.orchestra.run.smithi169.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-10-17T00:40:03.337 INFO:teuthology.orchestra.run.smithi169.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-10-17T00:40:03.340 INFO:teuthology.orchestra.run.smithi169.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-10-17T00:40:03.353 INFO:teuthology.orchestra.run.smithi169.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-10-17T00:40:03.355 INFO:teuthology.orchestra.run.smithi169.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-10-17T00:40:03.355 INFO:teuthology.orchestra.run.smithi169.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-10-17T00:40:03.358 INFO:teuthology.orchestra.run.smithi169.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-10-17T00:40:03.361 INFO:teuthology.orchestra.run.smithi169.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-10-17T00:40:03.365 INFO:teuthology.orchestra.run.smithi169.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-10-17T00:40:03.367 INFO:teuthology.orchestra.run.smithi169.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-10-17T00:40:03.370 INFO:teuthology.orchestra.run.smithi169.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-17T00:40:03.371 INFO:teuthology.orchestra.run.smithi169.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-10-17T00:40:03.371 INFO:teuthology.orchestra.run.smithi169.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-10-17T00:40:03.387 INFO:teuthology.orchestra.run.smithi169.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-10-17T00:40:03.388 INFO:teuthology.orchestra.run.smithi169.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-10-17T00:40:03.395 INFO:teuthology.orchestra.run.smithi169.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-10-17T00:40:03.395 INFO:teuthology.orchestra.run.smithi169.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-10-17T00:40:03.395 INFO:teuthology.orchestra.run.smithi169.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-10-17T00:40:03.397 INFO:teuthology.orchestra.run.smithi169.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-10-17T00:40:03.398 INFO:teuthology.orchestra.run.smithi169.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-10-17T00:40:03.583 INFO:teuthology.orchestra.run.smithi169.stdout:Get:80 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6740-gc3822f5f-1jammy [3,132 kB] 2023-10-17T00:40:03.788 INFO:teuthology.orchestra.run.smithi169.stdout:Get:81 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6740-gc3822f5f-1jammy [688 kB] 2023-10-17T00:40:03.830 INFO:teuthology.orchestra.run.smithi169.stdout:Get:82 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6740-gc3822f5f-1jammy [356 kB] 2023-10-17T00:40:03.860 INFO:teuthology.orchestra.run.smithi169.stdout:Get:83 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6740-gc3822f5f-1jammy [32.8 kB] 2023-10-17T00:40:03.861 INFO:teuthology.orchestra.run.smithi169.stdout:Get:84 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6740-gc3822f5f-1jammy [180 kB] 2023-10-17T00:40:03.865 INFO:teuthology.orchestra.run.smithi169.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6740-gc3822f5f-1jammy [62.3 kB] 2023-10-17T00:40:03.866 INFO:teuthology.orchestra.run.smithi169.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6740-gc3822f5f-1jammy [330 kB] 2023-10-17T00:40:03.893 INFO:teuthology.orchestra.run.smithi169.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6740-gc3822f5f-1jammy [6,221 kB] 2023-10-17T00:40:04.193 INFO:teuthology.orchestra.run.smithi169.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6740-gc3822f5f-1jammy [112 kB] 2023-10-17T00:40:04.195 INFO:teuthology.orchestra.run.smithi169.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6740-gc3822f5f-1jammy [439 kB] 2023-10-17T00:40:04.208 INFO:teuthology.orchestra.run.smithi169.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6740-gc3822f5f-1jammy [22.6 MB] 2023-10-17T00:40:05.061 INFO:teuthology.orchestra.run.smithi169.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6740-gc3822f5f-1jammy [6,853 kB] 2023-10-17T00:40:05.197 INFO:teuthology.orchestra.run.smithi169.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6740-gc3822f5f-1jammy [238 kB] 2023-10-17T00:40:05.202 INFO:teuthology.orchestra.run.smithi169.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6740-gc3822f5f-1jammy [128 kB] 2023-10-17T00:40:05.205 INFO:teuthology.orchestra.run.smithi169.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6740-gc3822f5f-1jammy [1,496 kB] 2023-10-17T00:40:05.254 INFO:teuthology.orchestra.run.smithi169.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6740-gc3822f5f-1jammy [9,863 kB] 2023-10-17T00:40:05.676 INFO:teuthology.orchestra.run.smithi169.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6740-gc3822f5f-1jammy [31.7 MB] 2023-10-17T00:40:06.822 INFO:teuthology.orchestra.run.smithi169.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6740-gc3822f5f-1jammy [14.0 kB] 2023-10-17T00:40:06.823 INFO:teuthology.orchestra.run.smithi169.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6740-gc3822f5f-1jammy [839 kB] 2023-10-17T00:40:06.839 INFO:teuthology.orchestra.run.smithi169.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6740-gc3822f5f-1jammy [2,146 kB] 2023-10-17T00:40:06.877 INFO:teuthology.orchestra.run.smithi169.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6740-gc3822f5f-1jammy [283 kB] 2023-10-17T00:40:06.882 INFO:teuthology.orchestra.run.smithi169.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6740-gc3822f5f-1jammy [138 kB] 2023-10-17T00:40:06.884 INFO:teuthology.orchestra.run.smithi169.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6740-gc3822f5f-1jammy [1,906 kB] 2023-10-17T00:40:06.906 INFO:teuthology.orchestra.run.smithi169.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6740-gc3822f5f-1jammy [8,625 kB] 2023-10-17T00:40:07.188 INFO:teuthology.orchestra.run.smithi169.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6740-gc3822f5f-1jammy [14.2 kB] 2023-10-17T00:40:07.189 INFO:teuthology.orchestra.run.smithi169.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6740-gc3822f5f-1jammy [47.8 MB] 2023-10-17T00:40:09.012 INFO:teuthology.orchestra.run.smithi169.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6740-gc3822f5f-1jammy [120 kB] 2023-10-17T00:40:09.015 INFO:teuthology.orchestra.run.smithi169.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6740-gc3822f5f-1jammy [38.3 kB] 2023-10-17T00:40:09.016 INFO:teuthology.orchestra.run.smithi169.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6740-gc3822f5f-1jammy [12.3 MB] 2023-10-17T00:40:09.326 INFO:teuthology.orchestra.run.smithi169.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6740-gc3822f5f-1jammy [91.4 kB] 2023-10-17T00:40:09.868 INFO:teuthology.orchestra.run.smithi169.stdout:Fetched 184 MB in 6s (28.5 MB/s) 2023-10-17T00:40:09.978 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package libfmt8:amd64. 2023-10-17T00:40:10.034 INFO:teuthology.orchestra.run.smithi169.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165988 files and directories currently installed.) 2023-10-17T00:40:10.039 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-10-17T00:40:10.111 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-17T00:40:10.439 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-10-17T00:40:10.466 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-10-17T00:40:10.506 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-17T00:40:10.868 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-17T00:40:10.894 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-10-17T00:40:10.934 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-17T00:40:11.299 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-17T00:40:11.326 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-10-17T00:40:11.370 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-17T00:40:11.816 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-17T00:40:11.842 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-17T00:40:11.891 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-17T00:40:12.277 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-17T00:40:12.303 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-17T00:40:12.335 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-17T00:40:12.680 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-17T00:40:12.707 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-17T00:40:12.747 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-17T00:40:13.117 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-10-17T00:40:13.143 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-10-17T00:40:13.183 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-17T00:40:13.461 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package librados2. 2023-10-17T00:40:13.484 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../008-librados2_18.0.0-6740-gc3822f5f-1jammy_amd64.deb ... 2023-10-17T00:40:13.519 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking librados2 (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:40:13.947 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package librbd1. 2023-10-17T00:40:13.973 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../009-librbd1_18.0.0-6740-gc3822f5f-1jammy_amd64.deb ... 2023-10-17T00:40:14.014 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking librbd1 (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:40:14.450 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package libcephfs2. 2023-10-17T00:40:14.476 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../010-libcephfs2_18.0.0-6740-gc3822f5f-1jammy_amd64.deb ... 2023-10-17T00:40:14.516 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking libcephfs2 (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:40:14.786 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-rados. 2023-10-17T00:40:14.812 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../011-python3-rados_18.0.0-6740-gc3822f5f-1jammy_amd64.deb ... 2023-10-17T00:40:14.852 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-rados (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:40:15.180 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-17T00:40:15.206 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.0.0-6740-gc3822f5f-1jammy_all.deb ... 2023-10-17T00:40:15.238 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-ceph-argparse (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:40:15.466 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-cephfs. 2023-10-17T00:40:15.481 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../013-python3-cephfs_18.0.0-6740-gc3822f5f-1jammy_amd64.deb ... 2023-10-17T00:40:15.524 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-cephfs (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:40:15.835 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-17T00:40:15.861 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../014-python3-ceph-common_18.0.0-6740-gc3822f5f-1jammy_all.deb ... 2023-10-17T00:40:15.902 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-ceph-common (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:40:16.230 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-17T00:40:16.256 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-10-17T00:40:16.305 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-17T00:40:16.608 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-prettytable. 2023-10-17T00:40:16.634 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2023-10-17T00:40:16.666 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-10-17T00:40:16.902 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-rbd. 2023-10-17T00:40:16.917 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../017-python3-rbd_18.0.0-6740-gc3822f5f-1jammy_amd64.deb ... 2023-10-17T00:40:16.960 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-rbd (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:40:17.313 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-17T00:40:17.339 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-10-17T00:40:17.388 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-17T00:40:17.766 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package librgw2. 2023-10-17T00:40:17.792 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../019-librgw2_18.0.0-6740-gc3822f5f-1jammy_amd64.deb ... 2023-10-17T00:40:17.833 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking librgw2 (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:40:18.327 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-rgw. 2023-10-17T00:40:18.342 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../020-python3-rgw_18.0.0-6740-gc3822f5f-1jammy_amd64.deb ... 2023-10-17T00:40:18.376 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-rgw (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:40:18.721 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package liboath0:amd64. 2023-10-17T00:40:18.747 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2023-10-17T00:40:18.788 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-10-17T00:40:19.124 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package libradosstriper1. 2023-10-17T00:40:19.150 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../022-libradosstriper1_18.0.0-6740-gc3822f5f-1jammy_amd64.deb ... 2023-10-17T00:40:19.190 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking libradosstriper1 (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:40:19.561 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package ceph-common. 2023-10-17T00:40:19.587 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../023-ceph-common_18.0.0-6740-gc3822f5f-1jammy_amd64.deb ... 2023-10-17T00:40:19.627 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking ceph-common (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:40:20.966 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package ceph-base. 2023-10-17T00:40:20.993 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../024-ceph-base_18.0.0-6740-gc3822f5f-1jammy_amd64.deb ... 2023-10-17T00:40:21.042 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking ceph-base (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:40:21.762 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-10-17T00:40:21.789 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-10-17T00:40:21.829 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-10-17T00:40:22.190 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-cheroot. 2023-10-17T00:40:22.217 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-10-17T00:40:22.265 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-17T00:40:22.635 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-10-17T00:40:22.661 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-10-17T00:40:22.710 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-10-17T00:40:23.104 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-jaraco.text. 2023-10-17T00:40:23.131 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2023-10-17T00:40:23.179 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-10-17T00:40:23.482 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-10-17T00:40:23.509 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-10-17T00:40:23.549 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-10-17T00:40:23.852 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-tempora. 2023-10-17T00:40:23.878 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2023-10-17T00:40:23.926 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-10-17T00:40:24.296 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-portend. 2023-10-17T00:40:24.322 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2023-10-17T00:40:24.371 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-10-17T00:40:24.699 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-10-17T00:40:24.725 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2023-10-17T00:40:24.766 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-10-17T00:40:25.068 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-17T00:40:25.095 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2023-10-17T00:40:25.135 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-10-17T00:40:25.505 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-natsort. 2023-10-17T00:40:25.531 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2023-10-17T00:40:25.572 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-10-17T00:40:25.883 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-logutils. 2023-10-17T00:40:25.910 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2023-10-17T00:40:25.950 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-10-17T00:40:26.219 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-17T00:40:26.245 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-10-17T00:40:26.277 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-10-17T00:40:26.556 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-mako. 2023-10-17T00:40:26.582 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-10-17T00:40:26.622 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-17T00:40:26.967 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-17T00:40:26.993 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-17T00:40:27.033 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-17T00:40:27.336 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-17T00:40:27.363 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-10-17T00:40:27.403 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-10-17T00:40:27.689 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-webob. 2023-10-17T00:40:27.704 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-10-17T00:40:27.739 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-10-17T00:40:28.050 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-waitress. 2023-10-17T00:40:28.077 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-10-17T00:40:28.112 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-17T00:40:28.378 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-tempita. 2023-10-17T00:40:28.404 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-10-17T00:40:28.445 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-10-17T00:40:28.756 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-paste. 2023-10-17T00:40:28.782 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-10-17T00:40:28.822 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-10-17T00:40:29.126 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-17T00:40:29.152 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-10-17T00:40:29.192 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-10-17T00:40:29.478 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-17T00:40:29.505 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2023-10-17T00:40:29.545 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-10-17T00:40:29.848 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-webtest. 2023-10-17T00:40:29.875 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2023-10-17T00:40:29.914 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-10-17T00:40:30.193 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-pecan. 2023-10-17T00:40:30.219 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-10-17T00:40:30.259 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-10-17T00:40:30.614 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-certifi. 2023-10-17T00:40:30.640 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2023-10-17T00:40:30.672 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-10-17T00:40:30.983 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-idna. 2023-10-17T00:40:31.010 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../049-python3-idna_3.3-1_all.deb ... 2023-10-17T00:40:31.050 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-idna (3.3-1) ... 2023-10-17T00:40:31.361 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-urllib3. 2023-10-17T00:40:31.387 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-10-17T00:40:31.427 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-10-17T00:40:31.747 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-requests. 2023-10-17T00:40:31.774 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-10-17T00:40:31.814 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-17T00:40:32.125 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-17T00:40:32.151 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-10-17T00:40:32.191 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-17T00:40:32.786 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-17T00:40:32.813 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.0.0-6740-gc3822f5f-1jammy_all.deb ... 2023-10-17T00:40:32.853 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:40:33.174 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-17T00:40:33.201 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.0.0-6740-gc3822f5f-1jammy_amd64.deb ... 2023-10-17T00:40:33.241 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:40:33.560 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-openssl. 2023-10-17T00:40:33.587 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../055-python3-openssl_21.0.0-1_all.deb ... 2023-10-17T00:40:33.627 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-10-17T00:40:33.972 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package ceph-mgr. 2023-10-17T00:40:33.998 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../056-ceph-mgr_18.0.0-6740-gc3822f5f-1jammy_amd64.deb ... 2023-10-17T00:40:34.038 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking ceph-mgr (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:40:34.425 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package ceph-mon. 2023-10-17T00:40:34.451 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../057-ceph-mon_18.0.0-6740-gc3822f5f-1jammy_amd64.deb ... 2023-10-17T00:40:34.483 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking ceph-mon (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:40:35.362 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package ceph-osd. 2023-10-17T00:40:35.388 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../058-ceph-osd_18.0.0-6740-gc3822f5f-1jammy_amd64.deb ... 2023-10-17T00:40:35.428 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking ceph-osd (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:40:37.865 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package ceph. 2023-10-17T00:40:37.880 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../059-ceph_18.0.0-6740-gc3822f5f-1jammy_amd64.deb ... 2023-10-17T00:40:37.923 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking ceph (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:40:38.226 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package ceph-fuse. 2023-10-17T00:40:38.252 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../060-ceph-fuse_18.0.0-6740-gc3822f5f-1jammy_amd64.deb ... 2023-10-17T00:40:38.284 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking ceph-fuse (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:40:38.637 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package ceph-mds. 2023-10-17T00:40:38.664 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../061-ceph-mds_18.0.0-6740-gc3822f5f-1jammy_amd64.deb ... 2023-10-17T00:40:38.703 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking ceph-mds (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:40:39.090 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package cephadm. 2023-10-17T00:40:39.117 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../062-cephadm_18.0.0-6740-gc3822f5f-1jammy_amd64.deb ... 2023-10-17T00:40:39.156 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking cephadm (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:40:39.443 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-asyncssh. 2023-10-17T00:40:39.469 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../063-python3-asyncssh_2.5.0-1_all.deb ... 2023-10-17T00:40:39.509 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-10-17T00:40:39.829 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-babel-localedata. 2023-10-17T00:40:39.855 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../064-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-10-17T00:40:39.896 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-17T00:40:40.909 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-babel. 2023-10-17T00:40:40.936 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../065-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-10-17T00:40:40.975 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-10-17T00:40:41.304 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-jinja2. 2023-10-17T00:40:41.331 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../066-python3-jinja2_3.0.3-1_all.deb ... 2023-10-17T00:40:41.370 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-10-17T00:40:41.732 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-17T00:40:41.759 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../067-ceph-mgr-cephadm_18.0.0-6740-gc3822f5f-1jammy_all.deb ... 2023-10-17T00:40:41.798 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:40:42.144 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-17T00:40:42.171 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../068-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-10-17T00:40:42.210 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-10-17T00:40:42.455 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-17T00:40:42.482 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../069-python3-repoze.lru_0.7-2_all.deb ... 2023-10-17T00:40:42.513 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-17T00:40:43.075 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-routes. 2023-10-17T00:40:43.102 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../070-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-10-17T00:40:43.141 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-10-17T00:40:43.469 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-17T00:40:43.496 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../071-ceph-mgr-dashboard_18.0.0-6740-gc3822f5f-1jammy_all.deb ... 2023-10-17T00:40:43.536 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:40:43.947 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-10-17T00:40:43.963 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../072-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-10-17T00:40:44.006 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-17T00:40:44.485 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-joblib. 2023-10-17T00:40:44.511 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../073-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-10-17T00:40:44.551 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-10-17T00:40:44.921 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-10-17T00:40:44.948 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../074-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-10-17T00:40:44.987 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-10-17T00:40:45.266 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-sklearn. 2023-10-17T00:40:45.293 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../075-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-10-17T00:40:45.324 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-17T00:40:46.033 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-17T00:40:46.060 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../076-ceph-mgr-diskprediction-local_18.0.0-6740-gc3822f5f-1jammy_all.deb ... 2023-10-17T00:40:46.099 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:40:47.044 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-cachetools. 2023-10-17T00:40:47.059 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../077-python3-cachetools_5.0.0-1_all.deb ... 2023-10-17T00:40:47.093 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-10-17T00:40:47.346 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-17T00:40:47.374 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../078-python3-pyasn1_0.4.8-1_all.deb ... 2023-10-17T00:40:47.413 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-10-17T00:40:47.744 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-17T00:40:47.771 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../079-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-10-17T00:40:47.810 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-10-17T00:40:48.121 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-rsa. 2023-10-17T00:40:48.149 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../080-python3-rsa_4.8-1_all.deb ... 2023-10-17T00:40:48.188 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-rsa (4.8-1) ... 2023-10-17T00:40:48.475 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-google-auth. 2023-10-17T00:40:48.502 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../081-python3-google-auth_1.5.1-3_all.deb ... 2023-10-17T00:40:48.533 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-10-17T00:40:48.803 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-17T00:40:48.819 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../082-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-10-17T00:40:48.853 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-17T00:40:49.249 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-websocket. 2023-10-17T00:40:49.276 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../083-python3-websocket_1.2.3-1_all.deb ... 2023-10-17T00:40:49.316 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-10-17T00:40:49.628 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-17T00:40:49.655 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../084-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-10-17T00:40:49.711 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-17T00:40:50.400 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-17T00:40:50.428 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../085-ceph-mgr-k8sevents_18.0.0-6740-gc3822f5f-1jammy_all.deb ... 2023-10-17T00:40:50.466 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:40:50.736 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-17T00:40:50.763 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../086-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-10-17T00:40:50.794 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-17T00:40:51.139 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-17T00:40:51.166 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../087-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-10-17T00:40:51.205 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-17T00:40:51.466 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package jq. 2023-10-17T00:40:51.482 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../088-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-10-17T00:40:51.524 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-10-17T00:40:51.769 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package socat. 2023-10-17T00:40:51.785 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../089-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-10-17T00:40:51.819 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-10-17T00:40:52.122 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package xmlstarlet. 2023-10-17T00:40:52.149 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../090-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-10-17T00:40:52.189 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-10-17T00:40:52.560 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package ceph-test. 2023-10-17T00:40:52.588 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../091-ceph-test_18.0.0-6740-gc3822f5f-1jammy_amd64.deb ... 2023-10-17T00:40:52.626 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking ceph-test (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:40:54.880 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package ceph-volume. 2023-10-17T00:40:54.896 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../092-ceph-volume_18.0.0-6740-gc3822f5f-1jammy_all.deb ... 2023-10-17T00:40:54.938 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking ceph-volume (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:40:55.243 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-17T00:40:55.259 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../093-libcephfs-dev_18.0.0-6740-gc3822f5f-1jammy_amd64.deb ... 2023-10-17T00:40:55.301 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking libcephfs-dev (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:40:55.671 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package nvme-cli. 2023-10-17T00:40:55.699 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../094-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-10-17T00:40:55.737 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-10-17T00:40:56.127 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-10-17T00:40:56.154 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../095-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-10-17T00:40:56.193 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-10-17T00:40:56.638 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-iniconfig. 2023-10-17T00:40:56.666 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../096-python3-iniconfig_1.1.1-2_all.deb ... 2023-10-17T00:40:56.704 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-10-17T00:40:57.007 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-pastescript. 2023-10-17T00:40:57.035 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../097-python3-pastescript_2.0.2-4_all.deb ... 2023-10-17T00:40:57.083 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-17T00:40:57.486 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-pluggy. 2023-10-17T00:40:57.513 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../098-python3-pluggy_0.13.0-7.1_all.deb ... 2023-10-17T00:40:57.561 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-10-17T00:40:57.939 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-psutil. 2023-10-17T00:40:57.966 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../099-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-10-17T00:40:58.014 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-10-17T00:40:58.750 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-py. 2023-10-17T00:40:58.778 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../100-python3-py_1.10.0-1_all.deb ... 2023-10-17T00:40:58.817 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-py (1.10.0-1) ... 2023-10-17T00:40:59.154 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-pygments. 2023-10-17T00:40:59.181 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../101-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-10-17T00:40:59.220 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-10-17T00:40:59.674 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-17T00:40:59.702 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../102-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-10-17T00:40:59.732 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-10-17T00:41:00.035 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-toml. 2023-10-17T00:41:00.063 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../103-python3-toml_0.10.2-1_all.deb ... 2023-10-17T00:41:00.102 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-10-17T00:41:00.405 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-pytest. 2023-10-17T00:41:00.432 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../104-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-10-17T00:41:00.471 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-10-17T00:41:00.800 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python3-simplejson. 2023-10-17T00:41:00.828 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-10-17T00:41:00.866 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-10-17T00:41:01.127 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-17T00:41:01.143 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-10-17T00:41:01.185 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-10-17T00:41:01.923 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package radosgw. 2023-10-17T00:41:01.951 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../107-radosgw_18.0.0-6740-gc3822f5f-1jammy_amd64.deb ... 2023-10-17T00:41:01.989 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking radosgw (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:41:02.751 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package rbd-fuse. 2023-10-17T00:41:02.767 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../108-rbd-fuse_18.0.0-6740-gc3822f5f-1jammy_amd64.deb ... 2023-10-17T00:41:02.809 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking rbd-fuse (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:41:03.113 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-10-17T00:41:03.356 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-17T00:41:03.482 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-10-17T00:41:03.745 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-10-17T00:41:04.113 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-10-17T00:41:04.624 INFO:teuthology.orchestra.run.smithi169.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-10-17T00:41:04.742 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up cephadm (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:41:05.260 INFO:teuthology.orchestra.run.smithi169.stdout:Adding system user cephadm....done 2023-10-17T00:41:05.424 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-17T00:41:05.733 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-10-17T00:41:05.984 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-10-17T00:41:06.127 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-10-17T00:41:06.378 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-17T00:41:06.638 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-10-17T00:41:06.789 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-py (1.10.0-1) ... 2023-10-17T00:41:07.107 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-10-17T00:41:07.468 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-10-17T00:41:07.711 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-10-17T00:41:07.987 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-10-17T00:41:08.255 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-10-17T00:41:08.522 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-10-17T00:41:08.790 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-ceph-argparse (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:41:09.083 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-17T00:41:09.201 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-10-17T00:41:09.460 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-17T00:41:09.578 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-10-17T00:41:09.954 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-17T00:41:10.105 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-10-17T00:41:10.398 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-10-17T00:41:10.682 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-10-17T00:41:10.984 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-10-17T00:41:11.669 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-10-17T00:41:11.937 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-10-17T00:41:12.063 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-10-17T00:41:12.189 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-17T00:41:12.515 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-10-17T00:41:12.925 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-10-17T00:41:13.168 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-10-17T00:41:13.544 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-17T00:41:13.846 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-17T00:41:14.239 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-10-17T00:41:14.532 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-10-17T00:41:14.658 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-ceph-common (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:41:14.951 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-idna (3.3-1) ... 2023-10-17T00:41:15.260 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-17T00:41:16.441 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-17T00:41:16.609 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-toml (0.10.2-1) ... 2023-10-17T00:41:16.868 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-17T00:41:16.995 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-10-17T00:41:17.146 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-10-17T00:41:17.464 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-10-17T00:41:17.740 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-10-17T00:41:17.991 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-17T00:41:18.126 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-17T00:41:18.252 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-10-17T00:41:18.553 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-10-17T00:41:18.813 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-10-17T00:41:19.114 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-10-17T00:41:19.365 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-17T00:41:19.624 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-10-17T00:41:19.867 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-17T00:41:19.976 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-10-17T00:41:20.261 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-17T00:41:20.387 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-17T00:41:20.730 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-10-17T00:41:21.056 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-10-17T00:41:21.332 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-10-17T00:41:21.745 INFO:teuthology.orchestra.run.smithi169.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-10-17T00:41:21.994 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-17T00:41:22.136 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-10-17T00:41:22.555 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-10-17T00:41:22.823 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-portend (3.0.0-1) ... 2023-10-17T00:41:23.115 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-17T00:41:23.233 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-10-17T00:41:23.551 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-10-17T00:41:23.852 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-17T00:41:24.145 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-10-17T00:41:24.296 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-rsa (4.8-1) ... 2023-10-17T00:41:24.564 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-10-17T00:41:24.840 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-17T00:41:25.099 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-10-17T00:41:25.468 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-17T00:41:25.761 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-10-17T00:41:26.113 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-17T00:41:26.239 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up librados2 (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:41:26.365 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up librgw2 (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:41:26.491 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:41:26.609 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up libcephfs2 (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:41:26.710 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up libradosstriper1 (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:41:26.921 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-10-17T00:41:27.198 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up librbd1 (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:41:27.324 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up ceph-mgr-modules-core (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:41:27.451 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up ceph-fuse (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:41:27.663 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-17T00:41:27.663 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-17T00:41:28.202 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up libcephfs-dev (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:41:28.353 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-rados (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:41:28.505 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-17T00:41:29.734 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-rbd (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:41:29.852 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up rbd-fuse (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:41:29.978 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-rgw (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:41:30.121 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python3-cephfs (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:41:30.272 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up ceph-common (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:41:30.848 INFO:teuthology.orchestra.run.smithi169.stdout:Adding group ceph....done 2023-10-17T00:41:31.216 INFO:teuthology.orchestra.run.smithi169.stdout:Adding system user ceph....done 2023-10-17T00:41:31.393 INFO:teuthology.orchestra.run.smithi169.stdout:Setting system user ceph properties....done 2023-10-17T00:41:31.412 INFO:teuthology.orchestra.run.smithi169.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-17T00:41:31.548 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-17T00:41:31.875 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-17T00:41:32.449 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up ceph-test (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:41:32.566 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up radosgw (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:41:33.002 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-17T00:41:33.003 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-17T00:41:33.502 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up ceph-base (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:41:33.857 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-17T00:41:34.370 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up ceph-mds (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:41:34.602 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-17T00:41:34.602 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-17T00:41:35.180 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up ceph-mgr (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:41:35.409 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-17T00:41:35.409 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-17T00:41:35.940 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up ceph-osd (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:41:36.246 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-17T00:41:36.246 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-17T00:41:36.767 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:41:36.876 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:41:37.027 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up ceph-mon (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:41:37.242 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-17T00:41:37.242 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-17T00:41:37.762 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up ceph-mgr-cephadm (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:41:37.921 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up ceph (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:41:38.047 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up ceph-mgr-dashboard (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:41:38.207 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up ceph-volume (18.0.0-6740-gc3822f5f-1jammy) ... 2023-10-17T00:41:38.518 INFO:teuthology.orchestra.run.smithi169.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-17T00:41:40.130 INFO:teuthology.orchestra.run.smithi169.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-17T00:41:40.598 INFO:teuthology.orchestra.run.smithi169.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-17T00:41:40.602 DEBUG:teuthology.parallel:result is None 2023-10-17T00:41:40.603 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=c3822f5f81e37dae2c0291e57dc72240858225fd 2023-10-17T00:41:40.754 DEBUG:teuthology.orchestra.run.smithi169:> dpkg-query -W -f '${Version}' ceph 2023-10-17T00:41:40.773 INFO:teuthology.orchestra.run.smithi169.stdout:18.0.0-6740-gc3822f5f-1jammy 2023-10-17T00:41:40.774 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6740-gc3822f5f-1jammy 2023-10-17T00:41:40.777 INFO:teuthology.task.install:The correct ceph version 18.0.0-6740-gc3822f5f-1jammy is installed. 2023-10-17T00:41:40.777 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-17T00:41:40.778 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-10-17T00:41:40.778 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-17T00:41:40.828 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-17T00:41:40.829 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-10-17T00:41:40.829 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/daemon-helper 2023-10-17T00:41:40.883 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-17T00:41:40.939 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-17T00:41:40.939 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-10-17T00:41:40.940 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-17T00:41:40.995 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-17T00:41:41.051 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-17T00:41:41.051 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-10-17T00:41:41.052 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/stdin-killer 2023-10-17T00:41:41.107 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-17T00:41:41.163 INFO:teuthology.run_tasks:Running task ceph... 2023-10-17T00:41:41.234 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-17T00:41:41.235 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 777 /var/log/ceph 2023-10-17T00:41:41.248 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-17T00:41:41.248 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-17T00:41:41.303 INFO:tasks.ceph:Creating extra log directories... 2023-10-17T00:41:41.303 DEBUG:teuthology.orchestra.run.smithi169:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-17T00:41:41.367 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-17T00:41:41.367 INFO:tasks.ceph:config {'conf': {'client': {'rbd default data pool': 'datapool', 'rbd default features': 61}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'debug rbd': 20}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-17T00:41:41.368 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rraja-2023-10-17_00:10:32-rbd:cli-debug-62994-distro-default-smithi/7430972', 'branch': 'debug-62994', 'description': 'rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/defaults msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery}', 'email': 'rraja@redhat.com', 'first_in_suite': False, 'job_id': '7430972', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'rraja-2023-10-17_00:10:32-rbd:cli-debug-62994-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'debug-62994'}, 'ceph': {'conf': {'client': {'rbd default data pool': 'datapool', 'rbd default features': 61}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'debug rbd': 20}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'c3822f5f81e37dae2c0291e57dc72240858225fd'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'c3822f5f81e37dae2c0291e57dc72240858225fd'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'debug-62994', 'sha1': 'c3822f5f81e37dae2c0291e57dc72240858225fd'}}, 'owner': 'scheduled_rraja@teuthology', 'priority': 50, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 9882, 'sha1': 'c3822f5f81e37dae2c0291e57dc72240858225fd', 'sleep_before_teardown': 0, 'suite': 'rbd:cli', 'suite_branch': 'debug-62994', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_c3822f5f81e37dae2c0291e57dc72240858225fd/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'c3822f5f81e37dae2c0291e57dc72240858225fd', 'targets': {'smithi169.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAaZyr1k69SkPlF/Xh4AZJUrxwBZlRxT2Tma5EXe7XCdHbtKN5MGae9jrpx30dbg96m2gGYzEUIuCckPOJylPVg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'exec': {'client.0': ['sudo ceph osd pool create datapool 4', 'rbd pool init datapool']}}, {'install': {'extra_system_packages': ['fio']}}, {'workunit': {'clients': {'client.0': ['rbd/rbd_support_module_recovery.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '8cdab074dcca9a68965bc5a50e9c30b691949723', 'timestamp': '2023-10-17_00:10:32', 'tube': 'smithi', 'user': 'rraja', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-17T00:41:41.368 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-17T00:41:41.408 DEBUG:teuthology.orchestra.run.smithi169:> sudo install -d -m0777 -- /var/run/ceph 2023-10-17T00:41:41.464 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-10-17T00:41:41.465 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/scratch_devs of=/dev/stdout 2023-10-17T00:41:41.512 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-17T00:41:41.512 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_1 2023-10-17T00:41:41.560 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-17T00:41:41.560 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-17T00:41:41.560 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 883 Links: 1 2023-10-17T00:41:41.560 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-17T00:41:41.561 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2023-10-17 00:36:38.392179435 +0000 2023-10-17T00:41:41.561 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2023-10-17 00:36:38.068182326 +0000 2023-10-17T00:41:41.561 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2023-10-17 00:36:38.068182326 +0000 2023-10-17T00:41:41.561 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: - 2023-10-17T00:41:41.562 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-17T00:41:41.614 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2023-10-17T00:41:41.614 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2023-10-17T00:41:41.614 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000444353 s, 1.2 MB/s 2023-10-17T00:41:41.615 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-17T00:41:41.666 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_2 2023-10-17T00:41:41.712 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-17T00:41:41.712 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-17T00:41:41.713 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 900 Links: 1 2023-10-17T00:41:41.713 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-17T00:41:41.713 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2023-10-17 00:36:39.012173904 +0000 2023-10-17T00:41:41.713 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2023-10-17 00:36:38.636177259 +0000 2023-10-17T00:41:41.713 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2023-10-17 00:36:38.636177259 +0000 2023-10-17T00:41:41.714 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: - 2023-10-17T00:41:41.714 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-17T00:41:41.767 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2023-10-17T00:41:41.767 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2023-10-17T00:41:41.767 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000403674 s, 1.3 MB/s 2023-10-17T00:41:41.768 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-17T00:41:41.818 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_3 2023-10-17T00:41:41.864 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-17T00:41:41.864 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-17T00:41:41.864 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 921 Links: 1 2023-10-17T00:41:41.864 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-17T00:41:41.865 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2023-10-17 00:36:39.664168089 +0000 2023-10-17T00:41:41.865 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2023-10-17 00:36:39.276171549 +0000 2023-10-17T00:41:41.865 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2023-10-17 00:36:39.276171549 +0000 2023-10-17T00:41:41.865 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: - 2023-10-17T00:41:41.866 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-17T00:41:41.918 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2023-10-17T00:41:41.918 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2023-10-17T00:41:41.919 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000367217 s, 1.4 MB/s 2023-10-17T00:41:41.920 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-17T00:41:41.969 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_4 2023-10-17T00:41:42.016 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-17T00:41:42.016 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-17T00:41:42.016 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 939 Links: 1 2023-10-17T00:41:42.016 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-17T00:41:42.017 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2023-10-17 00:36:40.276162629 +0000 2023-10-17T00:41:42.017 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2023-10-17 00:36:39.900165983 +0000 2023-10-17T00:41:42.017 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2023-10-17 00:36:39.900165983 +0000 2023-10-17T00:41:42.017 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: - 2023-10-17T00:41:42.017 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-17T00:41:42.070 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2023-10-17T00:41:42.071 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2023-10-17T00:41:42.071 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000391284 s, 1.3 MB/s 2023-10-17T00:41:42.072 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-17T00:41:42.121 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-17T00:41:42.121 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi169.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-17T00:41:42.122 INFO:tasks.ceph:Generating config... 2023-10-17T00:41:42.123 INFO:tasks.ceph:[client] rbd default data pool = datapool 2023-10-17T00:41:42.123 INFO:tasks.ceph:[client] rbd default features = 61 2023-10-17T00:41:42.124 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-10-17T00:41:42.124 INFO:tasks.ceph:[global] mon warn on pool no app = False 2023-10-17T00:41:42.124 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-10-17T00:41:42.124 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-17T00:41:42.125 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-17T00:41:42.125 INFO:tasks.ceph:[mgr] debug rbd = 20 2023-10-17T00:41:42.125 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-17T00:41:42.125 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-17T00:41:42.126 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-17T00:41:42.126 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-17T00:41:42.126 INFO:tasks.ceph:[osd] bluestore compression algorithm = zlib 2023-10-17T00:41:42.127 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-10-17T00:41:42.127 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-17T00:41:42.127 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-10-17T00:41:42.127 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-10-17T00:41:42.128 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-17T00:41:42.128 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-17T00:41:42.128 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-10-17T00:41:42.128 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-17T00:41:42.129 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-17T00:41:42.129 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-17T00:41:42.129 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-17T00:41:42.129 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-17T00:41:42.130 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-10-17T00:41:42.130 INFO:tasks.ceph:Setting up mon.a... 2023-10-17T00:41:42.130 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-17T00:41:42.195 INFO:teuthology.orchestra.run.smithi169.stdout:creating /etc/ceph/ceph.keyring 2023-10-17T00:41:42.208 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-17T00:41:42.283 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-17T00:41:42.339 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.169')] 2023-10-17T00:41:42.339 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.169', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug rbd': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'rbd default data pool': 'datapool', 'rbd default features': 61}, 'mon.a': {}} 2023-10-17T00:41:42.340 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-10-17T00:41:42.341 DEBUG:teuthology.orchestra.run.smithi169:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-17T00:41:42.388 DEBUG:teuthology.orchestra.run.smithi169:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.169 --print /home/ubuntu/cephtest/ceph.monmap 2023-10-17T00:41:42.467 INFO:teuthology.orchestra.run.smithi169.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-17T00:41:42.467 INFO:teuthology.orchestra.run.smithi169.stdout:monmaptool: generated fsid ce1c5384-f13f-4ca6-8231-3f754aa73d4f 2023-10-17T00:41:42.468 INFO:teuthology.orchestra.run.smithi169.stdout:setting min_mon_release = pacific 2023-10-17T00:41:42.468 INFO:teuthology.orchestra.run.smithi169.stdout:epoch 0 2023-10-17T00:41:42.468 INFO:teuthology.orchestra.run.smithi169.stdout:fsid ce1c5384-f13f-4ca6-8231-3f754aa73d4f 2023-10-17T00:41:42.468 INFO:teuthology.orchestra.run.smithi169.stdout:last_changed 2023-10-17T00:41:42.466497+0000 2023-10-17T00:41:42.468 INFO:teuthology.orchestra.run.smithi169.stdout:created 2023-10-17T00:41:42.466497+0000 2023-10-17T00:41:42.469 INFO:teuthology.orchestra.run.smithi169.stdout:min_mon_release 16 (pacific) 2023-10-17T00:41:42.469 INFO:teuthology.orchestra.run.smithi169.stdout:election_strategy: 1 2023-10-17T00:41:42.469 INFO:teuthology.orchestra.run.smithi169.stdout:0: [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] mon.a 2023-10-17T00:41:42.469 INFO:teuthology.orchestra.run.smithi169.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-10-17T00:41:42.470 DEBUG:teuthology.orchestra.run.smithi169:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-17T00:41:42.516 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID ce1c5384-f13f-4ca6-8231-3f754aa73d4f... 2023-10-17T00:41:42.517 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-17T00:41:42.589 INFO:teuthology.orchestra.run.smithi169.stdout:[global] 2023-10-17T00:41:42.589 INFO:teuthology.orchestra.run.smithi169.stdout: chdir = "" 2023-10-17T00:41:42.590 INFO:teuthology.orchestra.run.smithi169.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-17T00:41:42.590 INFO:teuthology.orchestra.run.smithi169.stdout: auth supported = cephx 2023-10-17T00:41:42.590 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:41:42.590 INFO:teuthology.orchestra.run.smithi169.stdout: filestore xattr use omap = true 2023-10-17T00:41:42.590 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:41:42.591 INFO:teuthology.orchestra.run.smithi169.stdout: mon clock drift allowed = 1.000 2023-10-17T00:41:42.591 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:41:42.591 INFO:teuthology.orchestra.run.smithi169.stdout: osd crush chooseleaf type = 0 2023-10-17T00:41:42.591 INFO:teuthology.orchestra.run.smithi169.stdout: auth debug = true 2023-10-17T00:41:42.591 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:41:42.592 INFO:teuthology.orchestra.run.smithi169.stdout: ms die on old message = true 2023-10-17T00:41:42.592 INFO:teuthology.orchestra.run.smithi169.stdout: ms die on bug = true 2023-10-17T00:41:42.592 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:41:42.592 INFO:teuthology.orchestra.run.smithi169.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-17T00:41:42.592 INFO:teuthology.orchestra.run.smithi169.stdout: mon pg warn max object skew = 0 2023-10-17T00:41:42.592 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:41:42.593 INFO:teuthology.orchestra.run.smithi169.stdout: # disable pg_autoscaler by default for new pools 2023-10-17T00:41:42.593 INFO:teuthology.orchestra.run.smithi169.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-17T00:41:42.593 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:41:42.593 INFO:teuthology.orchestra.run.smithi169.stdout: osd pool default size = 2 2023-10-17T00:41:42.593 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:41:42.594 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd allow primary affinity = true 2023-10-17T00:41:42.594 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd allow pg remap = true 2023-10-17T00:41:42.594 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on legacy crush tunables = false 2023-10-17T00:41:42.594 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on crush straw calc version zero = false 2023-10-17T00:41:42.594 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on no sortbitwise = false 2023-10-17T00:41:42.595 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on osd down out interval zero = false 2023-10-17T00:41:42.595 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on too few osds = false 2023-10-17T00:41:42.595 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-17T00:41:42.595 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-17T00:41:42.596 INFO:teuthology.orchestra.run.smithi169.stdout: mon_allow_pool_size_one = true 2023-10-17T00:41:42.596 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:41:42.596 INFO:teuthology.orchestra.run.smithi169.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-17T00:41:42.596 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:41:42.597 INFO:teuthology.orchestra.run.smithi169.stdout: osd default data pool replay window = 5 2023-10-17T00:41:42.597 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:41:42.597 INFO:teuthology.orchestra.run.smithi169.stdout: mon allow pool delete = true 2023-10-17T00:41:42.597 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:41:42.597 INFO:teuthology.orchestra.run.smithi169.stdout: mon cluster log file level = debug 2023-10-17T00:41:42.598 INFO:teuthology.orchestra.run.smithi169.stdout: debug asserts on shutdown = true 2023-10-17T00:41:42.598 INFO:teuthology.orchestra.run.smithi169.stdout: mon health detail to clog = false 2023-10-17T00:41:42.598 INFO:teuthology.orchestra.run.smithi169.stdout: mon host = 172.21.15.169 2023-10-17T00:41:42.598 INFO:teuthology.orchestra.run.smithi169.stdout: mon client directed command retry = 5 2023-10-17T00:41:42.598 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on pool no app = False 2023-10-17T00:41:42.598 INFO:teuthology.orchestra.run.smithi169.stdout: ms inject socket failures = 5000 2023-10-17T00:41:42.599 INFO:teuthology.orchestra.run.smithi169.stdout: fsid = ce1c5384-f13f-4ca6-8231-3f754aa73d4f 2023-10-17T00:41:42.599 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:41:42.599 INFO:teuthology.orchestra.run.smithi169.stdout:[osd] 2023-10-17T00:41:42.599 INFO:teuthology.orchestra.run.smithi169.stdout: osd journal size = 100 2023-10-17T00:41:42.599 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:41:42.600 INFO:teuthology.orchestra.run.smithi169.stdout: osd scrub load threshold = 5.0 2023-10-17T00:41:42.600 INFO:teuthology.orchestra.run.smithi169.stdout: osd scrub max interval = 600 2023-10-17T00:41:42.600 INFO:teuthology.orchestra.run.smithi169.stdout: osd mclock profile = high_recovery_ops 2023-10-17T00:41:42.600 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:41:42.600 INFO:teuthology.orchestra.run.smithi169.stdout: osd recover clone overlap = true 2023-10-17T00:41:42.600 INFO:teuthology.orchestra.run.smithi169.stdout: osd recovery max chunk = 1048576 2023-10-17T00:41:42.601 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:41:42.601 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug shutdown = true 2023-10-17T00:41:42.601 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug op order = true 2023-10-17T00:41:42.601 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug verify stray on activate = true 2023-10-17T00:41:42.601 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:41:42.601 INFO:teuthology.orchestra.run.smithi169.stdout: osd open classes on start = true 2023-10-17T00:41:42.602 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug pg log writeout = true 2023-10-17T00:41:42.602 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:41:42.602 INFO:teuthology.orchestra.run.smithi169.stdout: osd deep scrub update digest min age = 30 2023-10-17T00:41:42.602 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:41:42.602 INFO:teuthology.orchestra.run.smithi169.stdout: osd map max advance = 10 2023-10-17T00:41:42.603 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:41:42.603 INFO:teuthology.orchestra.run.smithi169.stdout: journal zero on create = true 2023-10-17T00:41:42.603 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:41:42.603 INFO:teuthology.orchestra.run.smithi169.stdout: filestore ondisk finisher threads = 3 2023-10-17T00:41:42.603 INFO:teuthology.orchestra.run.smithi169.stdout: filestore apply finisher threads = 3 2023-10-17T00:41:42.604 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:41:42.604 INFO:teuthology.orchestra.run.smithi169.stdout: bdev debug aio = true 2023-10-17T00:41:42.604 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug misdirected ops = true 2023-10-17T00:41:42.604 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore block size = 96636764160 2023-10-17T00:41:42.604 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore compression algorithm = zlib 2023-10-17T00:41:42.605 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore compression mode = aggressive 2023-10-17T00:41:42.605 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore fsck on mount = True 2023-10-17T00:41:42.605 INFO:teuthology.orchestra.run.smithi169.stdout: debug bluefs = 1/20 2023-10-17T00:41:42.605 INFO:teuthology.orchestra.run.smithi169.stdout: debug bluestore = 1/20 2023-10-17T00:41:42.605 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2023-10-17T00:41:42.605 INFO:teuthology.orchestra.run.smithi169.stdout: debug osd = 20 2023-10-17T00:41:42.606 INFO:teuthology.orchestra.run.smithi169.stdout: debug rocksdb = 4/10 2023-10-17T00:41:42.606 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd backfillfull_ratio = 0.85 2023-10-17T00:41:42.606 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd full ratio = 0.9 2023-10-17T00:41:42.606 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd nearfull ratio = 0.8 2023-10-17T00:41:42.607 INFO:teuthology.orchestra.run.smithi169.stdout: osd failsafe full ratio = 0.95 2023-10-17T00:41:42.607 INFO:teuthology.orchestra.run.smithi169.stdout: osd objectstore = bluestore 2023-10-17T00:41:42.607 INFO:teuthology.orchestra.run.smithi169.stdout: osd shutdown pgref assert = True 2023-10-17T00:41:42.607 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:41:42.607 INFO:teuthology.orchestra.run.smithi169.stdout:[mgr] 2023-10-17T00:41:42.608 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2023-10-17T00:41:42.608 INFO:teuthology.orchestra.run.smithi169.stdout: debug mgr = 20 2023-10-17T00:41:42.608 INFO:teuthology.orchestra.run.smithi169.stdout: debug mon = 20 2023-10-17T00:41:42.608 INFO:teuthology.orchestra.run.smithi169.stdout: debug auth = 20 2023-10-17T00:41:42.608 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min pgs per osd = 4 2023-10-17T00:41:42.608 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min bytes per osd = 10 2023-10-17T00:41:42.609 INFO:teuthology.orchestra.run.smithi169.stdout: mgr/telemetry/nag = false 2023-10-17T00:41:42.609 INFO:teuthology.orchestra.run.smithi169.stdout: debug rbd = 20 2023-10-17T00:41:42.609 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:41:42.609 INFO:teuthology.orchestra.run.smithi169.stdout:[mon] 2023-10-17T00:41:42.609 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2023-10-17T00:41:42.610 INFO:teuthology.orchestra.run.smithi169.stdout: debug mon = 20 2023-10-17T00:41:42.610 INFO:teuthology.orchestra.run.smithi169.stdout: debug paxos = 20 2023-10-17T00:41:42.610 INFO:teuthology.orchestra.run.smithi169.stdout: debug auth = 20 2023-10-17T00:41:42.610 INFO:teuthology.orchestra.run.smithi169.stdout: mon data avail warn = 5 2023-10-17T00:41:42.610 INFO:teuthology.orchestra.run.smithi169.stdout: mon mgr mkfs grace = 240 2023-10-17T00:41:42.611 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min pgs per osd = 4 2023-10-17T00:41:42.611 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd reporter subtree level = osd 2023-10-17T00:41:42.611 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd prime pg temp = true 2023-10-17T00:41:42.611 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min bytes per osd = 10 2023-10-17T00:41:42.611 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:41:42.611 INFO:teuthology.orchestra.run.smithi169.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-17T00:41:42.612 INFO:teuthology.orchestra.run.smithi169.stdout: auth mon ticket ttl = 660 # 11m 2023-10-17T00:41:42.612 INFO:teuthology.orchestra.run.smithi169.stdout: auth service ticket ttl = 240 # 4m 2023-10-17T00:41:42.612 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:41:42.612 INFO:teuthology.orchestra.run.smithi169.stdout: # don't complain about insecure global_id in the test suite 2023-10-17T00:41:42.612 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-17T00:41:42.613 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-17T00:41:42.613 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:41:42.613 INFO:teuthology.orchestra.run.smithi169.stdout: # 1m isn't quite enough 2023-10-17T00:41:42.613 INFO:teuthology.orchestra.run.smithi169.stdout: mon_down_mkfs_grace = 2m 2023-10-17T00:41:42.613 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:41:42.614 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_filestore_osds = false 2023-10-17T00:41:42.614 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:41:42.614 INFO:teuthology.orchestra.run.smithi169.stdout:[client] 2023-10-17T00:41:42.614 INFO:teuthology.orchestra.run.smithi169.stdout: rgw cache enabled = true 2023-10-17T00:41:42.614 INFO:teuthology.orchestra.run.smithi169.stdout: rgw enable ops log = true 2023-10-17T00:41:42.615 INFO:teuthology.orchestra.run.smithi169.stdout: rgw enable usage log = true 2023-10-17T00:41:42.615 INFO:teuthology.orchestra.run.smithi169.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-17T00:41:42.615 INFO:teuthology.orchestra.run.smithi169.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-17T00:41:42.615 INFO:teuthology.orchestra.run.smithi169.stdout: rbd default data pool = datapool 2023-10-17T00:41:42.615 INFO:teuthology.orchestra.run.smithi169.stdout: rbd default features = 61 2023-10-17T00:41:42.615 INFO:teuthology.orchestra.run.smithi169.stdout:[mon.a] 2023-10-17T00:41:42.617 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-17T00:41:42.617 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-17T00:41:42.709 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-17T00:41:42.709 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-10-17T00:41:42.709 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-17T00:41:42.715 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-10-17T00:41:42.715 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-17T00:41:42.759 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi169.front.sepia.ceph.com 2023-10-17T00:41:42.760 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-10-17T00:41:42.760 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-17T00:41:42.760 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-17T00:41:42.825 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-10-17T00:41:42.826 DEBUG:teuthology.orchestra.run.smithi169:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-17T00:41:42.872 INFO:tasks.ceph:Setting up mon nodes... 2023-10-17T00:41:42.872 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-17T00:41:42.873 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-17T00:41:42.958 INFO:teuthology.orchestra.run.smithi169.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-17T00:41:42.961 INFO:tasks.ceph:Setting up mds nodes... 2023-10-17T00:41:42.961 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-17T00:41:42.961 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-17T00:41:43.032 INFO:teuthology.orchestra.run.smithi169.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-17T00:41:43.041 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-17T00:41:43.042 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi169.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-17T00:41:43.042 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-17T00:41:43.095 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-17T00:41:43.095 INFO:tasks.ceph:role: osd.0 2023-10-17T00:41:43.095 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi169.front.sepia.ceph.com 2023-10-17T00:41:43.096 DEBUG:teuthology.orchestra.run.smithi169:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-17T00:41:43.164 INFO:teuthology.orchestra.run.smithi169.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-17T00:41:43.164 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-17T00:41:43.164 INFO:teuthology.orchestra.run.smithi169.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-17T00:41:43.164 INFO:teuthology.orchestra.run.smithi169.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-17T00:41:43.165 INFO:teuthology.orchestra.run.smithi169.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-17T00:41:43.165 INFO:teuthology.orchestra.run.smithi169.stdout: = sunit=0 swidth=0 blks 2023-10-17T00:41:43.165 INFO:teuthology.orchestra.run.smithi169.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-17T00:41:43.165 INFO:teuthology.orchestra.run.smithi169.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-17T00:41:43.165 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-17T00:41:43.166 INFO:teuthology.orchestra.run.smithi169.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-17T00:41:43.655 INFO:teuthology.orchestra.run.smithi169.stdout:Discarding blocks...Done. 2023-10-17T00:41:43.658 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi169.front.sepia.ceph.com -o noatime 2023-10-17T00:41:43.658 DEBUG:teuthology.orchestra.run.smithi169:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-17T00:41:43.798 DEBUG:teuthology.orchestra.run.smithi169:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-17T00:41:43.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-17T00:41:43.810 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: /sbin/restorecon: command not found 2023-10-17T00:41:43.810 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-17T00:41:43.863 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-17T00:41:43.863 INFO:tasks.ceph:role: osd.1 2023-10-17T00:41:43.863 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi169.front.sepia.ceph.com 2023-10-17T00:41:43.864 DEBUG:teuthology.orchestra.run.smithi169:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-17T00:41:43.927 INFO:teuthology.orchestra.run.smithi169.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-17T00:41:43.927 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-17T00:41:43.927 INFO:teuthology.orchestra.run.smithi169.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-17T00:41:43.928 INFO:teuthology.orchestra.run.smithi169.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-17T00:41:43.928 INFO:teuthology.orchestra.run.smithi169.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-17T00:41:43.928 INFO:teuthology.orchestra.run.smithi169.stdout: = sunit=0 swidth=0 blks 2023-10-17T00:41:43.928 INFO:teuthology.orchestra.run.smithi169.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-17T00:41:43.928 INFO:teuthology.orchestra.run.smithi169.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-17T00:41:43.929 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-17T00:41:43.929 INFO:teuthology.orchestra.run.smithi169.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-17T00:41:44.370 INFO:teuthology.orchestra.run.smithi169.stdout:Discarding blocks...Done. 2023-10-17T00:41:44.372 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi169.front.sepia.ceph.com -o noatime 2023-10-17T00:41:44.373 DEBUG:teuthology.orchestra.run.smithi169:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-17T00:41:44.464 DEBUG:teuthology.orchestra.run.smithi169:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-17T00:41:44.474 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: /sbin/restorecon: command not found 2023-10-17T00:41:44.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-17T00:41:44.475 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-17T00:41:44.529 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-17T00:41:44.530 INFO:tasks.ceph:role: osd.2 2023-10-17T00:41:44.530 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi169.front.sepia.ceph.com 2023-10-17T00:41:44.530 DEBUG:teuthology.orchestra.run.smithi169:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-17T00:41:44.591 INFO:teuthology.orchestra.run.smithi169.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-17T00:41:44.609 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-17T00:41:44.610 INFO:teuthology.orchestra.run.smithi169.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-17T00:41:44.610 INFO:teuthology.orchestra.run.smithi169.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-17T00:41:44.610 INFO:teuthology.orchestra.run.smithi169.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-17T00:41:44.610 INFO:teuthology.orchestra.run.smithi169.stdout: = sunit=0 swidth=0 blks 2023-10-17T00:41:44.611 INFO:teuthology.orchestra.run.smithi169.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-17T00:41:44.611 INFO:teuthology.orchestra.run.smithi169.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-17T00:41:44.611 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-17T00:41:44.611 INFO:teuthology.orchestra.run.smithi169.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-17T00:41:45.063 INFO:teuthology.orchestra.run.smithi169.stdout:Discarding blocks...Done. 2023-10-17T00:41:45.065 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi169.front.sepia.ceph.com -o noatime 2023-10-17T00:41:45.066 DEBUG:teuthology.orchestra.run.smithi169:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-17T00:41:45.143 DEBUG:teuthology.orchestra.run.smithi169:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-17T00:41:45.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-17T00:41:45.196 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: /sbin/restorecon: command not found 2023-10-17T00:41:45.197 DEBUG:teuthology.orchestra.run.smithi169:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-17T00:41:45.307 INFO:teuthology.orchestra.run.smithi169.stderr:2023-10-17T00:41:45.304+0000 7f99a40e9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-17T00:41:45.311 INFO:teuthology.orchestra.run.smithi169.stderr:2023-10-17T00:41:45.304+0000 7f99a40e9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-17T00:41:45.311 INFO:teuthology.orchestra.run.smithi169.stderr:2023-10-17T00:41:45.308+0000 7f99a40e9640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-17T00:41:45.311 INFO:teuthology.orchestra.run.smithi169.stderr:2023-10-17T00:41:45.308+0000 7f99a40e9640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-17T00:41:45.312 INFO:teuthology.orchestra.run.smithi169.stderr:2023-10-17T00:41:45.308+0000 7f99a40e9640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-17T00:41:45.319 INFO:teuthology.orchestra.run.smithi169.stderr:2023-10-17T00:41:45.312+0000 7f99a40e9640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-17T00:41:48.330 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-17T00:41:48.344 DEBUG:teuthology.orchestra.run.smithi169:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-17T00:41:48.426 INFO:teuthology.orchestra.run.smithi169.stderr:2023-10-17T00:41:48.420+0000 7fb75b413640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-17T00:41:48.428 INFO:teuthology.orchestra.run.smithi169.stderr:2023-10-17T00:41:48.424+0000 7fb75b413640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-17T00:41:48.428 INFO:teuthology.orchestra.run.smithi169.stderr:2023-10-17T00:41:48.424+0000 7fb75b413640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-17T00:41:48.428 INFO:teuthology.orchestra.run.smithi169.stderr:2023-10-17T00:41:48.424+0000 7fb75b413640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-17T00:41:48.429 INFO:teuthology.orchestra.run.smithi169.stderr:2023-10-17T00:41:48.424+0000 7fb75b413640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-17T00:41:48.429 INFO:teuthology.orchestra.run.smithi169.stderr:2023-10-17T00:41:48.424+0000 7fb75b413640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-17T00:41:51.446 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-17T00:41:51.499 DEBUG:teuthology.orchestra.run.smithi169:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-17T00:41:51.583 INFO:teuthology.orchestra.run.smithi169.stderr:2023-10-17T00:41:51.576+0000 7f5320510640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-17T00:41:51.583 INFO:teuthology.orchestra.run.smithi169.stderr:2023-10-17T00:41:51.580+0000 7f5320510640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-17T00:41:51.584 INFO:teuthology.orchestra.run.smithi169.stderr:2023-10-17T00:41:51.580+0000 7f5320510640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-17T00:41:51.584 INFO:teuthology.orchestra.run.smithi169.stderr:2023-10-17T00:41:51.580+0000 7f5320510640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-17T00:41:51.584 INFO:teuthology.orchestra.run.smithi169.stderr:2023-10-17T00:41:51.580+0000 7f5320510640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-17T00:41:51.584 INFO:teuthology.orchestra.run.smithi169.stderr:2023-10-17T00:41:51.580+0000 7f5320510640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-17T00:41:54.588 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-17T00:41:54.648 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-17T00:41:54.649 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-10-17T00:41:54.649 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-17T00:41:54.703 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-10-17T00:41:54.703 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-17T00:41:54.758 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-10-17T00:41:54.759 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-17T00:41:54.810 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-10-17T00:41:54.853 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-17T00:41:54.866 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-10-17T00:41:54.867 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-17T00:41:54.911 INFO:tasks.ceph:Adding keys to all mons... 2023-10-17T00:41:54.912 DEBUG:teuthology.orchestra.run.smithi169:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-17T00:41:54.966 INFO:teuthology.orchestra.run.smithi169.stdout:[mgr.x] 2023-10-17T00:41:54.966 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQBG2C1lN5UQORAAJwrwtbFRjwOVVcp65oIaxQ== 2023-10-17T00:41:54.967 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.0] 2023-10-17T00:41:54.967 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQBJ2C1lMXJMEhAADqm2+MVE7grwIGaS6746vw== 2023-10-17T00:41:54.967 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.1] 2023-10-17T00:41:54.967 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQBM2C1lUlNiGRAAX7aNKtV87PMoDPzcubgNdQ== 2023-10-17T00:41:54.968 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.2] 2023-10-17T00:41:54.968 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQBP2C1l+Fu1IhAAwUaMfK64rZS03zlKC4NVLg== 2023-10-17T00:41:54.968 INFO:teuthology.orchestra.run.smithi169.stdout:[client.0] 2023-10-17T00:41:54.968 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQBH2C1lvMvoARAA69QzwHecvARYr5qHVeUIoQ== 2023-10-17T00:41:54.969 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-17T00:41:55.056 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-17T00:41:55.099 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-17T00:41:55.186 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-17T00:41:55.266 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-17T00:41:55.348 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-17T00:41:55.349 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-17T00:41:55.400 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-17T00:41:55.490 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-17T00:41:55.501 DEBUG:teuthology.orchestra.run.smithi169:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-17T00:41:55.547 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-17T00:41:55.548 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-17T00:41:55.548 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-17T00:41:55.592 INFO:tasks.ceph.mon.a:Started 2023-10-17T00:41:55.593 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-17T00:41:55.593 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-17T00:41:55.593 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-17T00:41:55.596 INFO:tasks.ceph.mgr.x:Started 2023-10-17T00:41:55.596 INFO:tasks.ceph:Setting crush tunables to default 2023-10-17T00:41:55.596 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd crush tunables default 2023-10-17T00:41:55.840 INFO:teuthology.orchestra.run.smithi169.stderr:adjusted tunables profile to default 2023-10-17T00:41:55.856 INFO:tasks.ceph:check_enable_crimson: False 2023-10-17T00:41:55.856 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-17T00:41:55.857 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-10-17T00:41:55.857 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-17T00:41:55.867 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-10-17T00:41:55.868 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-17T00:41:55.920 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-10-17T00:41:55.921 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-17T00:41:55.978 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new 796660b8-af5d-47da-94d6-8e79f75895f7 0 2023-10-17T00:41:56.002 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:41:55.996+0000 7f3564073180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-17T00:41:56.200 INFO:teuthology.orchestra.run.smithi169.stdout:0 2023-10-17T00:41:56.218 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new f4c91e4d-51e9-4f95-a8c9-407c4d88298f 1 2023-10-17T00:41:56.413 INFO:teuthology.orchestra.run.smithi169.stdout:1 2023-10-17T00:41:56.431 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new f5e0fd93-fa0b-4c12-9996-f3e959ded07a 2 2023-10-17T00:41:56.560 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:41:56.556+0000 7f3564073180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-17T00:41:56.628 INFO:teuthology.orchestra.run.smithi169.stdout:2 2023-10-17T00:41:56.646 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-17T00:41:56.646 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-17T00:41:56.648 INFO:tasks.ceph.osd.0:Started 2023-10-17T00:41:56.649 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-17T00:41:56.649 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-17T00:41:56.651 INFO:tasks.ceph.osd.1:Started 2023-10-17T00:41:56.651 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-17T00:41:56.652 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-17T00:41:56.654 INFO:tasks.ceph.osd.2:Started 2023-10-17T00:41:56.654 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-17T00:41:56.750 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:41:56.744+0000 7f3564073180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-17T00:41:56.850 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:41:56.850 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":5,"fsid":"ce1c5384-f13f-4ca6-8231-3f754aa73d4f","created":"2023-10-17T00:41:55.707833+0000","modified":"2023-10-17T00:41:56.626469+0000","last_up_change":"0.000000","last_in_change":"2023-10-17T00:41:56.626469+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"796660b8-af5d-47da-94d6-8e79f75895f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f4c91e4d-51e9-4f95-a8c9-407c4d88298f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f5e0fd93-fa0b-4c12-9996-f3e959ded07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-17T00:41:56.871 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-17T00:41:56.872 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-17T00:41:57.174 DEBUG:teuthology.orchestra.run.smithi169:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-17T00:41:57.268 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:41:57.264+0000 7f31ed94a640 -1 Falling back to public interface 2023-10-17T00:41:57.269 INFO:tasks.ceph.osd.2.smithi169.stderr:2023-10-17T00:41:57.264+0000 7fb6bdc9a640 -1 Falling back to public interface 2023-10-17T00:41:57.269 INFO:tasks.ceph.osd.2.smithi169.stderr:2023-10-17T00:41:57.264+0000 7fb6bcc5b640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6805/0: (98) Address already in use 2023-10-17T00:41:57.270 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:41:57.264+0000 7f31ec90b640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6814/0: (98) Address already in use 2023-10-17T00:41:57.287 INFO:tasks.ceph.osd.0.smithi169.stderr:2023-10-17T00:41:57.284+0000 7f4fc5786640 -1 Falling back to public interface 2023-10-17T00:41:57.336 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:41:57.332+0000 7f3564073180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-17T00:41:57.345 INFO:teuthology.misc.health.smithi169.stdout: 2023-10-17T00:41:57.345 INFO:teuthology.misc.health.smithi169.stdout:{"epoch":5,"fsid":"ce1c5384-f13f-4ca6-8231-3f754aa73d4f","created":"2023-10-17T00:41:55.707833+0000","modified":"2023-10-17T00:41:56.626469+0000","last_up_change":"0.000000","last_in_change":"2023-10-17T00:41:56.626469+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"796660b8-af5d-47da-94d6-8e79f75895f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f4c91e4d-51e9-4f95-a8c9-407c4d88298f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f5e0fd93-fa0b-4c12-9996-f3e959ded07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-17T00:41:57.359 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-17T00:41:57.391 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:41:57.388+0000 7f3564073180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-17T00:41:57.444 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:41:57.440+0000 7f3564073180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-17T00:41:57.519 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:41:57.516+0000 7f3564073180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-17T00:41:57.571 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:41:57.564+0000 7f3564073180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-17T00:41:57.648 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:41:57.644+0000 7f3564073180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-17T00:41:57.699 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:41:57.692+0000 7f3564073180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-17T00:41:57.838 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:41:57.832+0000 7f3564073180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-17T00:41:57.897 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:41:57.891+0000 7f3564073180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-17T00:41:58.550 INFO:tasks.ceph.mgr.x.smithi169.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-10-17T00:41:58.550 INFO:tasks.ceph.mgr.x.smithi169.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-10-17T00:41:58.550 INFO:tasks.ceph.mgr.x.smithi169.stderr: from numpy import show_config as show_numpy_config 2023-10-17T00:41:58.612 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:41:58.607+0000 7f3564073180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-17T00:41:58.769 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:41:58.763+0000 7f3564073180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-17T00:41:58.940 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:41:58.935+0000 7f3564073180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-17T00:41:59.002 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:41:58.995+0000 7f3564073180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-17T00:41:59.125 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:41:59.119+0000 7f3564073180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-17T00:41:59.745 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:41:59.739+0000 7f3564073180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-17T00:41:59.820 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:41:59.815+0000 7f3564073180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-17T00:41:59.989 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:41:59.983+0000 7f3564073180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-17T00:42:00.068 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:42:00.063+0000 7f3564073180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-17T00:42:00.366 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:42:00.359+0000 7f3564073180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-17T00:42:00.448 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:42:00.443+0000 7f3564073180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-17T00:42:01.121 INFO:tasks.ceph.osd.2.smithi169.stderr:2023-10-17T00:42:01.115+0000 7fb6bdc9a640 -1 osd.2 0 log_to_monitors true 2023-10-17T00:42:01.167 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:42:01.163+0000 7f31ed94a640 -1 osd.1 0 log_to_monitors true 2023-10-17T00:42:01.233 INFO:tasks.ceph.osd.0.smithi169.stderr:2023-10-17T00:42:01.227+0000 7f4fc5786640 -1 osd.0 0 log_to_monitors true 2023-10-17T00:42:01.483 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:42:01.479+0000 7f350b8e2640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-17T00:42:03.661 DEBUG:teuthology.orchestra.run.smithi169:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-17T00:42:03.951 INFO:teuthology.misc.health.smithi169.stdout: 2023-10-17T00:42:03.952 INFO:teuthology.misc.health.smithi169.stdout:{"epoch":7,"fsid":"ce1c5384-f13f-4ca6-8231-3f754aa73d4f","created":"2023-10-17T00:41:55.707833+0000","modified":"2023-10-17T00:42:02.678327+0000","last_up_change":"0.000000","last_in_change":"2023-10-17T00:41:56.626469+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"796660b8-af5d-47da-94d6-8e79f75895f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f4c91e4d-51e9-4f95-a8c9-407c4d88298f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f5e0fd93-fa0b-4c12-9996-f3e959ded07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-17T00:42:03.969 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-17T00:42:05.744 INFO:tasks.ceph.osd.0.smithi169.stderr:2023-10-17T00:42:05.739+0000 7f4fc172e640 -1 osd.0 0 waiting for initial osdmap 2023-10-17T00:42:05.771 INFO:tasks.ceph.osd.0.smithi169.stderr:2023-10-17T00:42:05.767+0000 7f4fb9563640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-17T00:42:05.835 INFO:tasks.ceph.osd.2.smithi169.stderr:2023-10-17T00:42:05.831+0000 7fb6b9c42640 -1 osd.2 0 waiting for initial osdmap 2023-10-17T00:42:05.867 INFO:tasks.ceph.osd.2.smithi169.stderr:2023-10-17T00:42:05.863+0000 7fb6b1a77640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-17T00:42:05.905 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:42:05.899+0000 7f31e98f2640 -1 osd.1 0 waiting for initial osdmap 2023-10-17T00:42:05.931 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:42:05.927+0000 7f31e1727640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-17T00:42:10.271 DEBUG:teuthology.orchestra.run.smithi169:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-17T00:42:10.564 INFO:teuthology.misc.health.smithi169.stdout: 2023-10-17T00:42:10.564 INFO:teuthology.misc.health.smithi169.stdout:{"epoch":11,"fsid":"ce1c5384-f13f-4ca6-8231-3f754aa73d4f","created":"2023-10-17T00:41:55.707833+0000","modified":"2023-10-17T00:42:09.763212+0000","last_up_change":"2023-10-17T00:42:06.746971+0000","last_in_change":"2023-10-17T00:41:56.626469+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-17T00:42:08.546652+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"796660b8-af5d-47da-94d6-8e79f75895f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":2140921368},{"type":"v1","addr":"172.21.15.169:6817","nonce":2140921368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":2140921368},{"type":"v1","addr":"172.21.15.169:6819","nonce":2140921368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":2140921368},{"type":"v1","addr":"172.21.15.169:6823","nonce":2140921368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":2140921368},{"type":"v1","addr":"172.21.15.169:6821","nonce":2140921368}]},"public_addr":"172.21.15.169:6817/2140921368","cluster_addr":"172.21.15.169:6819/2140921368","heartbeat_back_addr":"172.21.15.169:6823/2140921368","heartbeat_front_addr":"172.21.15.169:6821/2140921368","state":["exists","up"]},{"osd":1,"uuid":"f4c91e4d-51e9-4f95-a8c9-407c4d88298f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":516036749},{"type":"v1","addr":"172.21.15.169:6801","nonce":516036749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":516036749},{"type":"v1","addr":"172.21.15.169:6805","nonce":516036749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":516036749},{"type":"v1","addr":"172.21.15.169:6815","nonce":516036749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":516036749},{"type":"v1","addr":"172.21.15.169:6810","nonce":516036749}]},"public_addr":"172.21.15.169:6801/516036749","cluster_addr":"172.21.15.169:6805/516036749","heartbeat_back_addr":"172.21.15.169:6815/516036749","heartbeat_front_addr":"172.21.15.169:6810/516036749","state":["exists","up"]},{"osd":2,"uuid":"f5e0fd93-fa0b-4c12-9996-f3e959ded07a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":1146691734},{"type":"v1","addr":"172.21.15.169:6803","nonce":1146691734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":1146691734},{"type":"v1","addr":"172.21.15.169:6807","nonce":1146691734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6813","nonce":1146691734},{"type":"v1","addr":"172.21.15.169:6814","nonce":1146691734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":1146691734},{"type":"v1","addr":"172.21.15.169:6811","nonce":1146691734}]},"public_addr":"172.21.15.169:6803/1146691734","cluster_addr":"172.21.15.169:6807/1146691734","heartbeat_back_addr":"172.21.15.169:6814/1146691734","heartbeat_front_addr":"172.21.15.169:6811/1146691734","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-17T00:42:02.258086+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-17T00:42:02.154164+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-17T00:42:02.110190+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-17T00:42:10.582 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-17T00:42:10.582 INFO:tasks.ceph:Creating RBD pool 2023-10-17T00:42:10.583 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-17T00:42:11.783 INFO:teuthology.orchestra.run.smithi169.stderr:pool 'rbd' created 2023-10-17T00:42:11.808 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-17T00:42:12.785 INFO:teuthology.orchestra.run.smithi169.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-17T00:42:12.814 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-17T00:42:12.815 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-17T00:42:12.815 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-17T00:42:12.816 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-17T00:42:12.816 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-17T00:42:13.134 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:42:13.151 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":5,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":2155024784},{"type":"v1","addr":"172.21.15.169:6825","nonce":2155024784}]},"active_addr":"172.21.15.169:6825/2155024784","active_change":"2023-10-17T00:42:00.467735+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.169:0","nonce":3578159913}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.169:0","nonce":3546353319}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.169:0","nonce":3228280117}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.169:0","nonce":3887334980}]}]} 2023-10-17T00:42:13.154 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-17T00:42:13.154 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-17T00:42:13.154 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-17T00:42:13.430 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:42:13.430 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":14,"fsid":"ce1c5384-f13f-4ca6-8231-3f754aa73d4f","created":"2023-10-17T00:41:55.707833+0000","modified":"2023-10-17T00:42:12.782584+0000","last_up_change":"2023-10-17T00:42:06.746971+0000","last_in_change":"2023-10-17T00:41:56.626469+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-17T00:42:08.546652+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-17T00:42:10.920165+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"796660b8-af5d-47da-94d6-8e79f75895f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":2140921368},{"type":"v1","addr":"172.21.15.169:6817","nonce":2140921368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":2140921368},{"type":"v1","addr":"172.21.15.169:6819","nonce":2140921368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":2140921368},{"type":"v1","addr":"172.21.15.169:6823","nonce":2140921368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":2140921368},{"type":"v1","addr":"172.21.15.169:6821","nonce":2140921368}]},"public_addr":"172.21.15.169:6817/2140921368","cluster_addr":"172.21.15.169:6819/2140921368","heartbeat_back_addr":"172.21.15.169:6823/2140921368","heartbeat_front_addr":"172.21.15.169:6821/2140921368","state":["exists","up"]},{"osd":1,"uuid":"f4c91e4d-51e9-4f95-a8c9-407c4d88298f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":516036749},{"type":"v1","addr":"172.21.15.169:6801","nonce":516036749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":516036749},{"type":"v1","addr":"172.21.15.169:6805","nonce":516036749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":516036749},{"type":"v1","addr":"172.21.15.169:6815","nonce":516036749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":516036749},{"type":"v1","addr":"172.21.15.169:6810","nonce":516036749}]},"public_addr":"172.21.15.169:6801/516036749","cluster_addr":"172.21.15.169:6805/516036749","heartbeat_back_addr":"172.21.15.169:6815/516036749","heartbeat_front_addr":"172.21.15.169:6810/516036749","state":["exists","up"]},{"osd":2,"uuid":"f5e0fd93-fa0b-4c12-9996-f3e959ded07a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":1146691734},{"type":"v1","addr":"172.21.15.169:6803","nonce":1146691734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":1146691734},{"type":"v1","addr":"172.21.15.169:6807","nonce":1146691734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6813","nonce":1146691734},{"type":"v1","addr":"172.21.15.169:6814","nonce":1146691734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":1146691734},{"type":"v1","addr":"172.21.15.169:6811","nonce":1146691734}]},"public_addr":"172.21.15.169:6803/1146691734","cluster_addr":"172.21.15.169:6807/1146691734","heartbeat_back_addr":"172.21.15.169:6814/1146691734","heartbeat_front_addr":"172.21.15.169:6811/1146691734","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-17T00:42:02.258086+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-17T00:42:02.154164+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-17T00:42:02.110190+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-17T00:42:13.448 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-17T00:42:13.449 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-17T00:42:13.731 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:42:13.731 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":14,"fsid":"ce1c5384-f13f-4ca6-8231-3f754aa73d4f","created":"2023-10-17T00:41:55.707833+0000","modified":"2023-10-17T00:42:12.782584+0000","last_up_change":"2023-10-17T00:42:06.746971+0000","last_in_change":"2023-10-17T00:41:56.626469+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-17T00:42:08.546652+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-17T00:42:10.920165+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"796660b8-af5d-47da-94d6-8e79f75895f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":2140921368},{"type":"v1","addr":"172.21.15.169:6817","nonce":2140921368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":2140921368},{"type":"v1","addr":"172.21.15.169:6819","nonce":2140921368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":2140921368},{"type":"v1","addr":"172.21.15.169:6823","nonce":2140921368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":2140921368},{"type":"v1","addr":"172.21.15.169:6821","nonce":2140921368}]},"public_addr":"172.21.15.169:6817/2140921368","cluster_addr":"172.21.15.169:6819/2140921368","heartbeat_back_addr":"172.21.15.169:6823/2140921368","heartbeat_front_addr":"172.21.15.169:6821/2140921368","state":["exists","up"]},{"osd":1,"uuid":"f4c91e4d-51e9-4f95-a8c9-407c4d88298f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":516036749},{"type":"v1","addr":"172.21.15.169:6801","nonce":516036749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":516036749},{"type":"v1","addr":"172.21.15.169:6805","nonce":516036749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":516036749},{"type":"v1","addr":"172.21.15.169:6815","nonce":516036749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":516036749},{"type":"v1","addr":"172.21.15.169:6810","nonce":516036749}]},"public_addr":"172.21.15.169:6801/516036749","cluster_addr":"172.21.15.169:6805/516036749","heartbeat_back_addr":"172.21.15.169:6815/516036749","heartbeat_front_addr":"172.21.15.169:6810/516036749","state":["exists","up"]},{"osd":2,"uuid":"f5e0fd93-fa0b-4c12-9996-f3e959ded07a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":1146691734},{"type":"v1","addr":"172.21.15.169:6803","nonce":1146691734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":1146691734},{"type":"v1","addr":"172.21.15.169:6807","nonce":1146691734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6813","nonce":1146691734},{"type":"v1","addr":"172.21.15.169:6814","nonce":1146691734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":1146691734},{"type":"v1","addr":"172.21.15.169:6811","nonce":1146691734}]},"public_addr":"172.21.15.169:6803/1146691734","cluster_addr":"172.21.15.169:6807/1146691734","heartbeat_back_addr":"172.21.15.169:6814/1146691734","heartbeat_front_addr":"172.21.15.169:6811/1146691734","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-17T00:42:02.258086+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-17T00:42:02.154164+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-17T00:42:02.110190+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-17T00:42:13.748 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-17T00:42:13.749 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-17T00:42:13.749 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-17T00:42:13.896 INFO:teuthology.orchestra.run.smithi169.stdout:34359738374 2023-10-17T00:42:13.897 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-17T00:42:13.912 INFO:teuthology.orchestra.run.smithi169.stdout:34359738373 2023-10-17T00:42:13.912 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-17T00:42:13.915 INFO:teuthology.orchestra.run.smithi169.stdout:34359738374 2023-10-17T00:42:13.916 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-17T00:42:14.172 INFO:teuthology.orchestra.run.smithi169.stdout:34359738372 2023-10-17T00:42:14.188 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738372 for osd.2 2023-10-17T00:42:14.230 INFO:teuthology.orchestra.run.smithi169.stdout:34359738372 2023-10-17T00:42:14.234 INFO:teuthology.orchestra.run.smithi169.stdout:34359738371 2023-10-17T00:42:14.250 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738372 for osd.0 2023-10-17T00:42:14.253 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738371 for osd.1 2023-10-17T00:42:15.189 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-17T00:42:15.251 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-17T00:42:15.253 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-17T00:42:15.486 INFO:teuthology.orchestra.run.smithi169.stdout:34359738374 2023-10-17T00:42:15.509 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738374 for osd.2 2023-10-17T00:42:15.510 DEBUG:teuthology.parallel:result is None 2023-10-17T00:42:15.547 INFO:teuthology.orchestra.run.smithi169.stdout:34359738374 2023-10-17T00:42:15.564 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738374 for osd.0 2023-10-17T00:42:15.564 DEBUG:teuthology.parallel:result is None 2023-10-17T00:42:15.572 INFO:teuthology.orchestra.run.smithi169.stdout:34359738373 2023-10-17T00:42:15.588 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.1 2023-10-17T00:42:15.588 DEBUG:teuthology.parallel:result is None 2023-10-17T00:42:15.589 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-17T00:42:15.589 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T00:42:15.895 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:42:15.896 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T00:42:15.911 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-10-17T00:42:14.474808+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81576,"kb_used_data":472,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033944,"statfs":{"total":289910292480,"available":289826758656,"internally_reserved":0,"allocated":483328,"data_stored":1281896,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.689332"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-17T00:42:12.798254+0000","last_change":"2023-10-17T00:42:12.798254+0000","last_active":"2023-10-17T00:42:12.798254+0000","last_peered":"2023-10-17T00:42:12.798254+0000","last_clean":"2023-10-17T00:42:12.798254+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T00:42:12.798254+0000","last_undegraded":"2023-10-17T00:42:12.798254+0000","last_fullsized":"2023-10-17T00:42:12.798254+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:01:24.297455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-17T00:42:12.800590+0000","last_change":"2023-10-17T00:42:12.800590+0000","last_active":"2023-10-17T00:42:12.800590+0000","last_peered":"2023-10-17T00:42:12.800590+0000","last_clean":"2023-10-17T00:42:12.800590+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T00:42:12.800590+0000","last_undegraded":"2023-10-17T00:42:12.800590+0000","last_fullsized":"2023-10-17T00:42:12.800590+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:57:09.886044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-17T00:42:12.798876+0000","last_change":"2023-10-17T00:42:12.798876+0000","last_active":"2023-10-17T00:42:12.798876+0000","last_peered":"2023-10-17T00:42:12.798876+0000","last_clean":"2023-10-17T00:42:12.798876+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T00:42:12.798876+0000","last_undegraded":"2023-10-17T00:42:12.798876+0000","last_fullsized":"2023-10-17T00:42:12.798876+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:01:36.228992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-17T00:42:12.796917+0000","last_change":"2023-10-17T00:42:12.796917+0000","last_active":"2023-10-17T00:42:12.796917+0000","last_peered":"2023-10-17T00:42:12.796917+0000","last_clean":"2023-10-17T00:42:12.796917+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T00:42:12.796917+0000","last_undegraded":"2023-10-17T00:42:12.796917+0000","last_fullsized":"2023-10-17T00:42:12.796917+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:02:16.539203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-17T00:42:12.797141+0000","last_change":"2023-10-17T00:42:12.797141+0000","last_active":"2023-10-17T00:42:12.797141+0000","last_peered":"2023-10-17T00:42:12.797141+0000","last_clean":"2023-10-17T00:42:12.797141+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T00:42:12.797141+0000","last_undegraded":"2023-10-17T00:42:12.797141+0000","last_fullsized":"2023-10-17T00:42:12.797141+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:49:46.671456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-17T00:42:13.793259+0000","last_change":"2023-10-17T00:42:12.799004+0000","last_active":"2023-10-17T00:42:13.793259+0000","last_peered":"2023-10-17T00:42:13.793259+0000","last_clean":"2023-10-17T00:42:13.793259+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T00:42:13.793259+0000","last_undegraded":"2023-10-17T00:42:13.793259+0000","last_fullsized":"2023-10-17T00:42:13.793259+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:21:48.553692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-17T00:42:13.793251+0000","last_change":"2023-10-17T00:42:12.799218+0000","last_active":"2023-10-17T00:42:13.793251+0000","last_peered":"2023-10-17T00:42:13.793251+0000","last_clean":"2023-10-17T00:42:13.793251+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T00:42:13.793251+0000","last_undegraded":"2023-10-17T00:42:13.793251+0000","last_fullsized":"2023-10-17T00:42:13.793251+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:00:25.745996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-17T00:42:12.796559+0000","last_change":"2023-10-17T00:42:12.796559+0000","last_active":"2023-10-17T00:42:12.796559+0000","last_peered":"2023-10-17T00:42:12.796559+0000","last_clean":"2023-10-17T00:42:12.796559+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T00:42:12.796559+0000","last_undegraded":"2023-10-17T00:42:12.796559+0000","last_fullsized":"2023-10-17T00:42:12.796559+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:31:47.957239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":140,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-17T00:42:12.693702+0000","last_change":"2023-10-17T00:42:10.044091+0000","last_active":"2023-10-17T00:42:12.693702+0000","last_peered":"2023-10-17T00:42:12.693702+0000","last_clean":"2023-10-17T00:42:12.693702+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T00:42:12.693702+0000","last_undegraded":"2023-10-17T00:42:12.693702+0000","last_fullsized":"2023-10-17T00:42:12.693702+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:08.753993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:08.753993+0000","last_clean_scrub_stamp":"2023-10-17T00:42:08.753993+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:18:21.220207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":122880,"data_stored":33720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82999999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":180224,"data_stored":624088,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":180224,"data_stored":624088,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T00:42:15.913 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T00:42:16.179 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:42:16.179 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T00:42:16.195 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-10-17T00:42:14.474808+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81576,"kb_used_data":472,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033944,"statfs":{"total":289910292480,"available":289826758656,"internally_reserved":0,"allocated":483328,"data_stored":1281896,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.689332"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-17T00:42:12.798254+0000","last_change":"2023-10-17T00:42:12.798254+0000","last_active":"2023-10-17T00:42:12.798254+0000","last_peered":"2023-10-17T00:42:12.798254+0000","last_clean":"2023-10-17T00:42:12.798254+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T00:42:12.798254+0000","last_undegraded":"2023-10-17T00:42:12.798254+0000","last_fullsized":"2023-10-17T00:42:12.798254+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:01:24.297455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-17T00:42:12.800590+0000","last_change":"2023-10-17T00:42:12.800590+0000","last_active":"2023-10-17T00:42:12.800590+0000","last_peered":"2023-10-17T00:42:12.800590+0000","last_clean":"2023-10-17T00:42:12.800590+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T00:42:12.800590+0000","last_undegraded":"2023-10-17T00:42:12.800590+0000","last_fullsized":"2023-10-17T00:42:12.800590+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:57:09.886044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-17T00:42:12.798876+0000","last_change":"2023-10-17T00:42:12.798876+0000","last_active":"2023-10-17T00:42:12.798876+0000","last_peered":"2023-10-17T00:42:12.798876+0000","last_clean":"2023-10-17T00:42:12.798876+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T00:42:12.798876+0000","last_undegraded":"2023-10-17T00:42:12.798876+0000","last_fullsized":"2023-10-17T00:42:12.798876+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:01:36.228992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-17T00:42:12.796917+0000","last_change":"2023-10-17T00:42:12.796917+0000","last_active":"2023-10-17T00:42:12.796917+0000","last_peered":"2023-10-17T00:42:12.796917+0000","last_clean":"2023-10-17T00:42:12.796917+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T00:42:12.796917+0000","last_undegraded":"2023-10-17T00:42:12.796917+0000","last_fullsized":"2023-10-17T00:42:12.796917+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:02:16.539203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-17T00:42:12.797141+0000","last_change":"2023-10-17T00:42:12.797141+0000","last_active":"2023-10-17T00:42:12.797141+0000","last_peered":"2023-10-17T00:42:12.797141+0000","last_clean":"2023-10-17T00:42:12.797141+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T00:42:12.797141+0000","last_undegraded":"2023-10-17T00:42:12.797141+0000","last_fullsized":"2023-10-17T00:42:12.797141+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:49:46.671456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-17T00:42:13.793259+0000","last_change":"2023-10-17T00:42:12.799004+0000","last_active":"2023-10-17T00:42:13.793259+0000","last_peered":"2023-10-17T00:42:13.793259+0000","last_clean":"2023-10-17T00:42:13.793259+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T00:42:13.793259+0000","last_undegraded":"2023-10-17T00:42:13.793259+0000","last_fullsized":"2023-10-17T00:42:13.793259+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:21:48.553692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-17T00:42:13.793251+0000","last_change":"2023-10-17T00:42:12.799218+0000","last_active":"2023-10-17T00:42:13.793251+0000","last_peered":"2023-10-17T00:42:13.793251+0000","last_clean":"2023-10-17T00:42:13.793251+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T00:42:13.793251+0000","last_undegraded":"2023-10-17T00:42:13.793251+0000","last_fullsized":"2023-10-17T00:42:13.793251+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:00:25.745996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-17T00:42:12.796559+0000","last_change":"2023-10-17T00:42:12.796559+0000","last_active":"2023-10-17T00:42:12.796559+0000","last_peered":"2023-10-17T00:42:12.796559+0000","last_clean":"2023-10-17T00:42:12.796559+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T00:42:12.796559+0000","last_undegraded":"2023-10-17T00:42:12.796559+0000","last_fullsized":"2023-10-17T00:42:12.796559+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:31:47.957239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":140,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-17T00:42:12.693702+0000","last_change":"2023-10-17T00:42:10.044091+0000","last_active":"2023-10-17T00:42:12.693702+0000","last_peered":"2023-10-17T00:42:12.693702+0000","last_clean":"2023-10-17T00:42:12.693702+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T00:42:12.693702+0000","last_undegraded":"2023-10-17T00:42:12.693702+0000","last_fullsized":"2023-10-17T00:42:12.693702+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:08.753993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:08.753993+0000","last_clean_scrub_stamp":"2023-10-17T00:42:08.753993+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:18:21.220207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":122880,"data_stored":33720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82999999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":180224,"data_stored":624088,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":180224,"data_stored":624088,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T00:42:16.196 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-17T00:42:16.196 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-17T00:42:16.196 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-17T00:42:16.197 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-17T00:42:16.490 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T00:42:16.491 INFO:teuthology.orchestra.run.smithi169.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-17T00:42:16.507 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-17T00:42:16.508 INFO:teuthology.run_tasks:Running task exec... 2023-10-17T00:42:16.518 INFO:teuthology.task.exec:Executing custom commands... 2023-10-17T00:42:16.518 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi169.front.sepia.ceph.com 2023-10-17T00:42:16.519 DEBUG:teuthology.orchestra.run.smithi169:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool create datapool 4' 2023-10-17T00:42:17.809 INFO:teuthology.orchestra.run.smithi169.stderr:pool 'datapool' created 2023-10-17T00:42:17.835 DEBUG:teuthology.orchestra.run.smithi169:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'rbd pool init datapool' 2023-10-17T00:42:20.917 INFO:teuthology.run_tasks:Running task install... 2023-10-17T00:42:20.927 DEBUG:teuthology.task.install:project ceph 2023-10-17T00:42:20.927 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c3822f5f81e37dae2c0291e57dc72240858225fd'}} 2023-10-17T00:42:20.927 DEBUG:teuthology.task.install:config {'extra_system_packages': ['fio'], 'flavor': 'default', 'sha1': 'c3822f5f81e37dae2c0291e57dc72240858225fd'} 2023-10-17T00:42:20.928 INFO:teuthology.task.install:Using flavor: default 2023-10-17T00:42:20.933 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-17T00:42:20.934 INFO:teuthology.task.install:extra packages: [] 2023-10-17T00:42:20.934 DEBUG:teuthology.orchestra.run.smithi169:> sudo apt-key list | grep Ceph 2023-10-17T00:42:20.997 INFO:teuthology.orchestra.run.smithi169.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-17T00:42:21.026 INFO:teuthology.orchestra.run.smithi169.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-17T00:42:21.026 INFO:teuthology.orchestra.run.smithi169.stdout:uid [ unknown] Ceph.com (release key) 2023-10-17T00:42:21.027 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-10-17T00:42:21.028 INFO:teuthology.task.install.deb:Installing system (non-project) packages: fio on remote deb x86_64 2023-10-17T00:42:21.028 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=c3822f5f81e37dae2c0291e57dc72240858225fd 2023-10-17T00:42:21.355 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default/ 2023-10-17T00:42:21.355 INFO:teuthology.task.install.deb:Package version is 18.0.0-6740-gc3822f5f-1jammy 2023-10-17T00:42:21.646 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-10-17T00:42:21.646 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-17T00:42:21.661 DEBUG:teuthology.orchestra.run.smithi169:> sudo apt-get update 2023-10-17T00:42:21.945 INFO:teuthology.orchestra.run.smithi169.stdout:Ign:1 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy InRelease 2023-10-17T00:42:21.977 INFO:teuthology.orchestra.run.smithi169.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-17T00:42:21.978 INFO:teuthology.orchestra.run.smithi169.stdout:Hit:3 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-17T00:42:21.981 INFO:teuthology.orchestra.run.smithi169.stdout:Hit:4 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy Release 2023-10-17T00:42:22.033 INFO:teuthology.orchestra.run.smithi169.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/debug-62994/c3822f5f81e37dae2c0291e57dc72240858225fd/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-17T00:42:22.049 INFO:teuthology.orchestra.run.smithi169.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-17T00:42:22.130 INFO:teuthology.orchestra.run.smithi169.stdout:Hit:7 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-17T00:42:23.242 INFO:teuthology.orchestra.run.smithi169.stdout:Reading package lists... 2023-10-17T00:42:23.257 DEBUG:teuthology.orchestra.run.smithi169:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6740-gc3822f5f-1jammy cephadm=18.0.0-6740-gc3822f5f-1jammy ceph-mds=18.0.0-6740-gc3822f5f-1jammy ceph-mgr=18.0.0-6740-gc3822f5f-1jammy ceph-common=18.0.0-6740-gc3822f5f-1jammy ceph-fuse=18.0.0-6740-gc3822f5f-1jammy ceph-test=18.0.0-6740-gc3822f5f-1jammy ceph-volume=18.0.0-6740-gc3822f5f-1jammy radosgw=18.0.0-6740-gc3822f5f-1jammy python3-rados=18.0.0-6740-gc3822f5f-1jammy python3-rgw=18.0.0-6740-gc3822f5f-1jammy python3-cephfs=18.0.0-6740-gc3822f5f-1jammy python3-rbd=18.0.0-6740-gc3822f5f-1jammy libcephfs2=18.0.0-6740-gc3822f5f-1jammy libcephfs-dev=18.0.0-6740-gc3822f5f-1jammy librados2=18.0.0-6740-gc3822f5f-1jammy librbd1=18.0.0-6740-gc3822f5f-1jammy rbd-fuse=18.0.0-6740-gc3822f5f-1jammy 2023-10-17T00:42:23.318 INFO:teuthology.orchestra.run.smithi169.stdout:Reading package lists... 2023-10-17T00:42:23.503 INFO:teuthology.orchestra.run.smithi169.stdout:Building dependency tree... 2023-10-17T00:42:23.503 INFO:teuthology.orchestra.run.smithi169.stdout:Reading state information... 2023-10-17T00:42:23.669 INFO:teuthology.orchestra.run.smithi169.stdout:ceph is already the newest version (18.0.0-6740-gc3822f5f-1jammy). 2023-10-17T00:42:23.669 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-common is already the newest version (18.0.0-6740-gc3822f5f-1jammy). 2023-10-17T00:42:23.669 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-fuse is already the newest version (18.0.0-6740-gc3822f5f-1jammy). 2023-10-17T00:42:23.669 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-mds is already the newest version (18.0.0-6740-gc3822f5f-1jammy). 2023-10-17T00:42:23.670 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-mgr is already the newest version (18.0.0-6740-gc3822f5f-1jammy). 2023-10-17T00:42:23.670 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-test is already the newest version (18.0.0-6740-gc3822f5f-1jammy). 2023-10-17T00:42:23.670 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-volume is already the newest version (18.0.0-6740-gc3822f5f-1jammy). 2023-10-17T00:42:23.670 INFO:teuthology.orchestra.run.smithi169.stdout:cephadm is already the newest version (18.0.0-6740-gc3822f5f-1jammy). 2023-10-17T00:42:23.670 INFO:teuthology.orchestra.run.smithi169.stdout:libcephfs-dev is already the newest version (18.0.0-6740-gc3822f5f-1jammy). 2023-10-17T00:42:23.671 INFO:teuthology.orchestra.run.smithi169.stdout:libcephfs2 is already the newest version (18.0.0-6740-gc3822f5f-1jammy). 2023-10-17T00:42:23.671 INFO:teuthology.orchestra.run.smithi169.stdout:librados2 is already the newest version (18.0.0-6740-gc3822f5f-1jammy). 2023-10-17T00:42:23.671 INFO:teuthology.orchestra.run.smithi169.stdout:librbd1 is already the newest version (18.0.0-6740-gc3822f5f-1jammy). 2023-10-17T00:42:23.671 INFO:teuthology.orchestra.run.smithi169.stdout:python3-cephfs is already the newest version (18.0.0-6740-gc3822f5f-1jammy). 2023-10-17T00:42:23.671 INFO:teuthology.orchestra.run.smithi169.stdout:python3-rados is already the newest version (18.0.0-6740-gc3822f5f-1jammy). 2023-10-17T00:42:23.671 INFO:teuthology.orchestra.run.smithi169.stdout:python3-rbd is already the newest version (18.0.0-6740-gc3822f5f-1jammy). 2023-10-17T00:42:23.672 INFO:teuthology.orchestra.run.smithi169.stdout:python3-rgw is already the newest version (18.0.0-6740-gc3822f5f-1jammy). 2023-10-17T00:42:23.672 INFO:teuthology.orchestra.run.smithi169.stdout:radosgw is already the newest version (18.0.0-6740-gc3822f5f-1jammy). 2023-10-17T00:42:23.672 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-fuse is already the newest version (18.0.0-6740-gc3822f5f-1jammy). 2023-10-17T00:42:23.672 INFO:teuthology.orchestra.run.smithi169.stdout:The following packages were automatically installed and are no longer required: 2023-10-17T00:42:23.672 INFO:teuthology.orchestra.run.smithi169.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-17T00:42:23.673 INFO:teuthology.orchestra.run.smithi169.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-17T00:42:23.673 INFO:teuthology.orchestra.run.smithi169.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-17T00:42:23.673 INFO:teuthology.orchestra.run.smithi169.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-17T00:42:23.673 INFO:teuthology.orchestra.run.smithi169.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-17T00:42:23.673 INFO:teuthology.orchestra.run.smithi169.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-17T00:42:23.707 INFO:teuthology.orchestra.run.smithi169.stdout:0 upgraded, 0 newly installed, 0 to remove and 256 not upgraded. 2023-10-17T00:42:23.708 INFO:teuthology.orchestra.run.smithi169.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-17T00:42:23.709 DEBUG:teuthology.orchestra.run.smithi169:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install fio 2023-10-17T00:42:23.768 INFO:teuthology.orchestra.run.smithi169.stdout:Reading package lists... 2023-10-17T00:42:23.954 INFO:teuthology.orchestra.run.smithi169.stdout:Building dependency tree... 2023-10-17T00:42:23.955 INFO:teuthology.orchestra.run.smithi169.stdout:Reading state information... 2023-10-17T00:42:24.116 INFO:teuthology.orchestra.run.smithi169.stdout:The following packages were automatically installed and are no longer required: 2023-10-17T00:42:24.117 INFO:teuthology.orchestra.run.smithi169.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libiscsi7 libpmemobj1 2023-10-17T00:42:24.117 INFO:teuthology.orchestra.run.smithi169.stdout: libpython2-dev libpython2-stdlib libpython2.7 libpython2.7-dev 2023-10-17T00:42:24.117 INFO:teuthology.orchestra.run.smithi169.stdout: libpython2.7-minimal libpython2.7-stdlib linux-image-5.4.0-135-generic 2023-10-17T00:42:24.117 INFO:teuthology.orchestra.run.smithi169.stdout: linux-modules-5.4.0-135-generic python2 python2-dev python2-minimal 2023-10-17T00:42:24.118 INFO:teuthology.orchestra.run.smithi169.stdout: python2.7 python2.7-dev python2.7-minimal 2023-10-17T00:42:24.118 INFO:teuthology.orchestra.run.smithi169.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-17T00:42:24.119 INFO:teuthology.orchestra.run.smithi169.stdout:The following additional packages will be installed: 2023-10-17T00:42:24.119 INFO:teuthology.orchestra.run.smithi169.stdout: libpmemblk1 2023-10-17T00:42:24.119 INFO:teuthology.orchestra.run.smithi169.stdout:Suggested packages: 2023-10-17T00:42:24.120 INFO:teuthology.orchestra.run.smithi169.stdout: gnuplot gfio python-scipy 2023-10-17T00:42:24.144 INFO:teuthology.orchestra.run.smithi169.stdout:The following NEW packages will be installed: 2023-10-17T00:42:24.145 INFO:teuthology.orchestra.run.smithi169.stdout: fio libpmemblk1 2023-10-17T00:42:24.212 INFO:teuthology.orchestra.run.smithi169.stdout:0 upgraded, 2 newly installed, 0 to remove and 256 not upgraded. 2023-10-17T00:42:24.212 INFO:teuthology.orchestra.run.smithi169.stdout:Need to get 4,112 kB of archives. 2023-10-17T00:42:24.213 INFO:teuthology.orchestra.run.smithi169.stdout:After this operation, 7,813 kB of additional disk space will be used. 2023-10-17T00:42:24.213 INFO:teuthology.orchestra.run.smithi169.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libpmemblk1 amd64 1.11.1-3build1 [65.6 kB] 2023-10-17T00:42:24.359 INFO:teuthology.orchestra.run.smithi169.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 fio amd64 3.28-1 [4,047 kB] 2023-10-17T00:42:25.011 INFO:teuthology.orchestra.run.smithi169.stdout:Fetched 4,112 kB in 1s (8,078 kB/s) 2023-10-17T00:42:25.179 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package libpmemblk1:amd64. 2023-10-17T00:42:25.283 INFO:teuthology.orchestra.run.smithi169.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 172814 files and directories currently installed.) 2023-10-17T00:42:25.289 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../libpmemblk1_1.11.1-3build1_amd64.deb ... 2023-10-17T00:42:25.369 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking libpmemblk1:amd64 (1.11.1-3build1) ... 2023-10-17T00:42:25.839 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package fio. 2023-10-17T00:42:25.867 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../archives/fio_3.28-1_amd64.deb ... 2023-10-17T00:42:25.906 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking fio (3.28-1) ... 2023-10-17T00:42:26.549 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up libpmemblk1:amd64 (1.11.1-3build1) ... 2023-10-17T00:42:26.669 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up fio (3.28-1) ... 2023-10-17T00:42:27.206 INFO:teuthology.orchestra.run.smithi169.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-17T00:42:28.323 INFO:teuthology.orchestra.run.smithi169.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-17T00:42:28.801 INFO:teuthology.orchestra.run.smithi169.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-17T00:42:28.803 DEBUG:teuthology.parallel:result is None 2023-10-17T00:42:28.804 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=c3822f5f81e37dae2c0291e57dc72240858225fd 2023-10-17T00:42:28.942 DEBUG:teuthology.orchestra.run.smithi169:> dpkg-query -W -f '${Version}' ceph 2023-10-17T00:42:28.963 INFO:teuthology.orchestra.run.smithi169.stdout:18.0.0-6740-gc3822f5f-1jammy 2023-10-17T00:42:28.963 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6740-gc3822f5f-1jammy 2023-10-17T00:42:28.963 INFO:teuthology.task.install:The correct ceph version 18.0.0-6740-gc3822f5f-1jammy is installed. 2023-10-17T00:42:28.964 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2023-10-17T00:42:28.964 INFO:teuthology.run_tasks:Running task workunit... 2023-10-17T00:42:28.977 INFO:tasks.workunit:Pulling workunits from ref c3822f5f81e37dae2c0291e57dc72240858225fd 2023-10-17T00:42:28.978 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-10-17T00:42:28.979 DEBUG:teuthology.orchestra.run.smithi169:> stat -- /home/ubuntu/cephtest/mnt.0 2023-10-17T00:42:29.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-17T00:42:29.012 INFO:teuthology.orchestra.run.smithi169.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-10-17T00:42:29.013 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-10-17T00:42:29.059 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-10-17T00:42:29.060 DEBUG:teuthology.orchestra.run.smithi169:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-10-17T00:42:29.107 INFO:tasks.workunit:timeout=3h 2023-10-17T00:42:29.107 INFO:tasks.workunit:cleanup=True 2023-10-17T00:42:29.108 DEBUG:teuthology.orchestra.run.smithi169:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout c3822f5f81e37dae2c0291e57dc72240858225fd 2023-10-17T00:42:29.157 INFO:tasks.workunit.client.0.smithi169.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-10-17T00:43:28.907 INFO:tasks.workunit.client.0.smithi169.stderr:Note: switching to 'c3822f5f81e37dae2c0291e57dc72240858225fd'. 2023-10-17T00:43:28.908 INFO:tasks.workunit.client.0.smithi169.stderr: 2023-10-17T00:43:28.908 INFO:tasks.workunit.client.0.smithi169.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-10-17T00:43:28.908 INFO:tasks.workunit.client.0.smithi169.stderr:changes and commit them, and you can discard any commits you make in this 2023-10-17T00:43:28.908 INFO:tasks.workunit.client.0.smithi169.stderr:state without impacting any branches by switching back to a branch. 2023-10-17T00:43:28.909 INFO:tasks.workunit.client.0.smithi169.stderr: 2023-10-17T00:43:28.909 INFO:tasks.workunit.client.0.smithi169.stderr:If you want to create a new branch to retain commits you create, you may 2023-10-17T00:43:28.909 INFO:tasks.workunit.client.0.smithi169.stderr:do so (now or later) by using -c with the switch command. Example: 2023-10-17T00:43:28.909 INFO:tasks.workunit.client.0.smithi169.stderr: 2023-10-17T00:43:28.909 INFO:tasks.workunit.client.0.smithi169.stderr: git switch -c 2023-10-17T00:43:28.910 INFO:tasks.workunit.client.0.smithi169.stderr: 2023-10-17T00:43:28.910 INFO:tasks.workunit.client.0.smithi169.stderr:Or undo this operation with: 2023-10-17T00:43:28.911 INFO:tasks.workunit.client.0.smithi169.stderr: 2023-10-17T00:43:28.911 INFO:tasks.workunit.client.0.smithi169.stderr: git switch - 2023-10-17T00:43:28.911 INFO:tasks.workunit.client.0.smithi169.stderr: 2023-10-17T00:43:28.911 INFO:tasks.workunit.client.0.smithi169.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-10-17T00:43:28.911 INFO:tasks.workunit.client.0.smithi169.stderr: 2023-10-17T00:43:28.911 INFO:tasks.workunit.client.0.smithi169.stderr:HEAD is now at c3822f5f81e qa/suites/rbd: add test to check rbd_support module recovery 2023-10-17T00:43:28.913 DEBUG:teuthology.orchestra.run.smithi169:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-10-17T00:43:28.959 INFO:tasks.workunit.client.0.smithi169.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-10-17T00:43:28.962 INFO:tasks.workunit.client.0.smithi169.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-17T00:43:28.962 INFO:tasks.workunit.client.0.smithi169.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-10-17T00:43:29.715 INFO:tasks.workunit.client.0.smithi169.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-10-17T00:43:29.807 INFO:tasks.workunit.client.0.smithi169.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-10-17T00:43:29.860 INFO:tasks.workunit.client.0.smithi169.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-17T00:43:29.862 INFO:tasks.workunit.client.0.smithi169.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-17T00:43:29.862 INFO:tasks.workunit.client.0.smithi169.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-10-17T00:43:29.909 INFO:tasks.workunit.client.0.smithi169.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-17T00:43:29.912 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-10-17T00:43:29.912 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-10-17T00:43:29.958 INFO:tasks.workunit:Running workunits matching rbd/rbd_support_module_recovery.sh on client.0... 2023-10-17T00:43:29.959 INFO:tasks.workunit:Running workunit rbd/rbd_support_module_recovery.sh... 2023-10-17T00:43:29.960 DEBUG:teuthology.orchestra.run.smithi169:workunit test rbd/rbd_support_module_recovery.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c3822f5f81e37dae2c0291e57dc72240858225fd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh 2023-10-17T00:43:30.010 INFO:tasks.workunit.client.0.smithi169.stderr:+ POOL=rbd 2023-10-17T00:43:30.010 INFO:tasks.workunit.client.0.smithi169.stderr:+ IMAGE_PREFIX=image 2023-10-17T00:43:30.011 INFO:tasks.workunit.client.0.smithi169.stderr:+ NUM_IMAGES=20 2023-10-17T00:43:30.011 INFO:tasks.workunit.client.0.smithi169.stderr:+ RUN_TIME=3600 2023-10-17T00:43:30.011 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror pool enable rbd image 2023-10-17T00:43:30.054 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror pool peer add rbd dummy 2023-10-17T00:43:30.090 INFO:tasks.workunit.client.0.smithi169.stdout:a78f389e-4449-4cbc-9a22-09a6a1a0ef53 2023-10-17T00:43:30.093 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i = 1 )) 2023-10-17T00:43:30.093 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:30.093 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image1 2023-10-17T00:43:30.136 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror image enable rbd/image1 snapshot 2023-10-17T00:43:30.788 INFO:tasks.workunit.client.0.smithi169.stdout:Mirroring enabled 2023-10-17T00:43:30.801 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 1m 2023-10-17T00:43:30.863 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:30.863 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:30.864 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image2 2023-10-17T00:43:30.914 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror image enable rbd/image2 snapshot 2023-10-17T00:43:31.788 INFO:tasks.workunit.client.0.smithi169.stdout:Mirroring enabled 2023-10-17T00:43:31.801 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image2 1m 2023-10-17T00:43:31.863 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:31.863 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:31.863 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image3 2023-10-17T00:43:31.915 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror image enable rbd/image3 snapshot 2023-10-17T00:43:32.791 INFO:tasks.workunit.client.0.smithi169.stdout:Mirroring enabled 2023-10-17T00:43:32.805 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image3 1m 2023-10-17T00:43:32.866 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:32.866 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:32.866 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image4 2023-10-17T00:43:32.915 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror image enable rbd/image4 snapshot 2023-10-17T00:43:33.814 INFO:tasks.workunit.client.0.smithi169.stdout:Mirroring enabled 2023-10-17T00:43:33.830 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image4 1m 2023-10-17T00:43:33.893 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:33.893 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:33.893 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image5 2023-10-17T00:43:33.940 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror image enable rbd/image5 snapshot 2023-10-17T00:43:34.817 INFO:tasks.workunit.client.0.smithi169.stdout:Mirroring enabled 2023-10-17T00:43:34.827 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image5 1m 2023-10-17T00:43:34.888 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:34.888 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:34.889 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image6 2023-10-17T00:43:34.937 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror image enable rbd/image6 snapshot 2023-10-17T00:43:35.826 INFO:tasks.workunit.client.0.smithi169.stdout:Mirroring enabled 2023-10-17T00:43:35.836 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image6 1m 2023-10-17T00:43:35.897 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:35.897 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:35.897 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image7 2023-10-17T00:43:35.944 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror image enable rbd/image7 snapshot 2023-10-17T00:43:36.830 INFO:tasks.workunit.client.0.smithi169.stdout:Mirroring enabled 2023-10-17T00:43:36.844 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image7 1m 2023-10-17T00:43:36.905 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:36.905 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:36.905 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image8 2023-10-17T00:43:36.956 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror image enable rbd/image8 snapshot 2023-10-17T00:43:37.842 INFO:tasks.workunit.client.0.smithi169.stdout:Mirroring enabled 2023-10-17T00:43:37.856 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image8 1m 2023-10-17T00:43:37.922 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:37.923 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:37.923 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image9 2023-10-17T00:43:37.970 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror image enable rbd/image9 snapshot 2023-10-17T00:43:38.850 INFO:tasks.workunit.client.0.smithi169.stdout:Mirroring enabled 2023-10-17T00:43:38.864 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image9 1m 2023-10-17T00:43:38.926 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:38.926 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:38.927 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image10 2023-10-17T00:43:38.973 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror image enable rbd/image10 snapshot 2023-10-17T00:43:39.864 INFO:tasks.workunit.client.0.smithi169.stdout:Mirroring enabled 2023-10-17T00:43:39.878 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image10 1m 2023-10-17T00:43:39.943 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:39.943 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:39.944 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image11 2023-10-17T00:43:39.991 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror image enable rbd/image11 snapshot 2023-10-17T00:43:40.856 INFO:tasks.workunit.client.0.smithi169.stdout:Mirroring enabled 2023-10-17T00:43:40.870 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image11 1m 2023-10-17T00:43:40.931 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:40.931 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:40.932 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image12 2023-10-17T00:43:40.978 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror image enable rbd/image12 snapshot 2023-10-17T00:43:41.872 INFO:tasks.workunit.client.0.smithi169.stdout:Mirroring enabled 2023-10-17T00:43:41.884 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image12 1m 2023-10-17T00:43:41.945 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:41.945 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:41.945 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image13 2023-10-17T00:43:41.995 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror image enable rbd/image13 snapshot 2023-10-17T00:43:42.884 INFO:tasks.workunit.client.0.smithi169.stdout:Mirroring enabled 2023-10-17T00:43:42.897 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image13 1m 2023-10-17T00:43:42.957 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:42.957 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:42.957 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image14 2023-10-17T00:43:43.007 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror image enable rbd/image14 snapshot 2023-10-17T00:43:43.887 INFO:tasks.workunit.client.0.smithi169.stdout:Mirroring enabled 2023-10-17T00:43:43.900 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image14 1m 2023-10-17T00:43:43.959 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:43.959 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:43.959 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image15 2023-10-17T00:43:44.007 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror image enable rbd/image15 snapshot 2023-10-17T00:43:44.892 INFO:tasks.workunit.client.0.smithi169.stdout:Mirroring enabled 2023-10-17T00:43:44.906 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image15 1m 2023-10-17T00:43:44.968 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:44.968 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:44.968 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image16 2023-10-17T00:43:45.015 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror image enable rbd/image16 snapshot 2023-10-17T00:43:45.908 INFO:tasks.workunit.client.0.smithi169.stdout:Mirroring enabled 2023-10-17T00:43:45.922 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image16 1m 2023-10-17T00:43:45.984 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:45.985 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:45.985 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image17 2023-10-17T00:43:46.033 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror image enable rbd/image17 snapshot 2023-10-17T00:43:46.912 INFO:tasks.workunit.client.0.smithi169.stdout:Mirroring enabled 2023-10-17T00:43:46.927 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image17 1m 2023-10-17T00:43:46.990 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:46.990 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:46.990 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image18 2023-10-17T00:43:47.037 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror image enable rbd/image18 snapshot 2023-10-17T00:43:47.920 INFO:tasks.workunit.client.0.smithi169.stdout:Mirroring enabled 2023-10-17T00:43:47.934 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image18 1m 2023-10-17T00:43:47.997 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:47.998 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:47.998 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image19 2023-10-17T00:43:48.050 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror image enable rbd/image19 snapshot 2023-10-17T00:43:48.930 INFO:tasks.workunit.client.0.smithi169.stdout:Mirroring enabled 2023-10-17T00:43:48.944 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image19 1m 2023-10-17T00:43:49.005 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:49.006 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:49.006 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image20 2023-10-17T00:43:49.053 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror image enable rbd/image20 snapshot 2023-10-17T00:43:49.939 INFO:tasks.workunit.client.0.smithi169.stdout:Mirroring enabled 2023-10-17T00:43:49.950 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image20 1m 2023-10-17T00:43:50.012 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:50.012 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:50.012 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i = 1 )) 2023-10-17T00:43:50.012 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:50.013 INFO:tasks.workunit.client.0.smithi169.stderr:++ sudo rbd device map rbd/image1 2023-10-17T00:43:50.217 INFO:tasks.workunit.client.0.smithi169.stderr:+ DEVS[$i]=/dev/rbd0 2023-10-17T00:43:50.217 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:50.217 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:50.217 INFO:tasks.workunit.client.0.smithi169.stderr:+ fio --name=fiotest --filename=/dev/rbd0 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-17T00:43:50.218 INFO:tasks.workunit.client.0.smithi169.stderr:++ sudo rbd device map rbd/image2 2023-10-17T00:43:50.288 INFO:tasks.workunit.client.0.smithi169.stderr:+ DEVS[$i]=/dev/rbd1 2023-10-17T00:43:50.288 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:50.288 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:50.289 INFO:tasks.workunit.client.0.smithi169.stderr:+ fio --name=fiotest --filename=/dev/rbd1 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-17T00:43:50.289 INFO:tasks.workunit.client.0.smithi169.stderr:++ sudo rbd device map rbd/image3 2023-10-17T00:43:50.360 INFO:tasks.workunit.client.0.smithi169.stderr:+ DEVS[$i]=/dev/rbd2 2023-10-17T00:43:50.360 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:50.361 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:50.361 INFO:tasks.workunit.client.0.smithi169.stderr:+ fio --name=fiotest --filename=/dev/rbd2 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-17T00:43:50.361 INFO:tasks.workunit.client.0.smithi169.stderr:++ sudo rbd device map rbd/image4 2023-10-17T00:43:50.436 INFO:tasks.workunit.client.0.smithi169.stderr:+ DEVS[$i]=/dev/rbd3 2023-10-17T00:43:50.436 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:50.436 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:50.436 INFO:tasks.workunit.client.0.smithi169.stderr:+ fio --name=fiotest --filename=/dev/rbd3 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-17T00:43:50.437 INFO:tasks.workunit.client.0.smithi169.stderr:++ sudo rbd device map rbd/image5 2023-10-17T00:43:50.509 INFO:tasks.workunit.client.0.smithi169.stderr:+ DEVS[$i]=/dev/rbd4 2023-10-17T00:43:50.509 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:50.510 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:50.510 INFO:tasks.workunit.client.0.smithi169.stderr:+ fio --name=fiotest --filename=/dev/rbd4 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-17T00:43:50.510 INFO:tasks.workunit.client.0.smithi169.stderr:++ sudo rbd device map rbd/image6 2023-10-17T00:43:50.598 INFO:tasks.workunit.client.0.smithi169.stderr:+ DEVS[$i]=/dev/rbd5 2023-10-17T00:43:50.598 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:50.598 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:50.599 INFO:tasks.workunit.client.0.smithi169.stderr:+ fio --name=fiotest --filename=/dev/rbd5 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-17T00:43:50.599 INFO:tasks.workunit.client.0.smithi169.stderr:++ sudo rbd device map rbd/image7 2023-10-17T00:43:50.693 INFO:tasks.workunit.client.0.smithi169.stderr:+ DEVS[$i]=/dev/rbd6 2023-10-17T00:43:50.694 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:50.694 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:50.694 INFO:tasks.workunit.client.0.smithi169.stderr:+ fio --name=fiotest --filename=/dev/rbd6 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-17T00:43:50.694 INFO:tasks.workunit.client.0.smithi169.stderr:++ sudo rbd device map rbd/image8 2023-10-17T00:43:50.802 INFO:tasks.workunit.client.0.smithi169.stderr:+ DEVS[$i]=/dev/rbd7 2023-10-17T00:43:50.802 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:50.803 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:50.803 INFO:tasks.workunit.client.0.smithi169.stderr:+ fio --name=fiotest --filename=/dev/rbd7 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-17T00:43:50.803 INFO:tasks.workunit.client.0.smithi169.stderr:++ sudo rbd device map rbd/image9 2023-10-17T00:43:50.927 INFO:tasks.workunit.client.0.smithi169.stderr:+ DEVS[$i]=/dev/rbd8 2023-10-17T00:43:50.927 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:50.927 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:50.928 INFO:tasks.workunit.client.0.smithi169.stderr:+ fio --name=fiotest --filename=/dev/rbd8 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-17T00:43:50.928 INFO:tasks.workunit.client.0.smithi169.stderr:++ sudo rbd device map rbd/image10 2023-10-17T00:43:51.064 INFO:tasks.workunit.client.0.smithi169.stderr:+ DEVS[$i]=/dev/rbd9 2023-10-17T00:43:51.065 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:51.065 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:51.065 INFO:tasks.workunit.client.0.smithi169.stderr:+ fio --name=fiotest --filename=/dev/rbd9 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-17T00:43:51.065 INFO:tasks.workunit.client.0.smithi169.stderr:++ sudo rbd device map rbd/image11 2023-10-17T00:43:51.167 INFO:tasks.workunit.client.0.smithi169.stderr:+ DEVS[$i]=/dev/rbd10 2023-10-17T00:43:51.167 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:51.167 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:51.167 INFO:tasks.workunit.client.0.smithi169.stderr:+ fio --name=fiotest --filename=/dev/rbd10 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-17T00:43:51.168 INFO:tasks.workunit.client.0.smithi169.stderr:++ sudo rbd device map rbd/image12 2023-10-17T00:43:51.278 INFO:tasks.workunit.client.0.smithi169.stderr:+ DEVS[$i]=/dev/rbd11 2023-10-17T00:43:51.278 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:51.279 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:51.279 INFO:tasks.workunit.client.0.smithi169.stderr:+ fio --name=fiotest --filename=/dev/rbd11 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-17T00:43:51.279 INFO:tasks.workunit.client.0.smithi169.stderr:++ sudo rbd device map rbd/image13 2023-10-17T00:43:51.413 INFO:tasks.workunit.client.0.smithi169.stderr:+ DEVS[$i]=/dev/rbd12 2023-10-17T00:43:51.413 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:51.414 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:51.414 INFO:tasks.workunit.client.0.smithi169.stderr:+ fio --name=fiotest --filename=/dev/rbd12 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-17T00:43:51.414 INFO:tasks.workunit.client.0.smithi169.stderr:++ sudo rbd device map rbd/image14 2023-10-17T00:43:51.514 INFO:tasks.workunit.client.0.smithi169.stderr:+ DEVS[$i]=/dev/rbd13 2023-10-17T00:43:51.515 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:51.515 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:51.515 INFO:tasks.workunit.client.0.smithi169.stderr:++ sudo rbd device map rbd/image15 2023-10-17T00:43:51.515 INFO:tasks.workunit.client.0.smithi169.stderr:+ fio --name=fiotest --filename=/dev/rbd13 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-17T00:43:51.659 INFO:tasks.workunit.client.0.smithi169.stderr:+ DEVS[$i]=/dev/rbd14 2023-10-17T00:43:51.660 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:51.660 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:51.660 INFO:tasks.workunit.client.0.smithi169.stderr:++ sudo rbd device map rbd/image16 2023-10-17T00:43:51.665 INFO:tasks.workunit.client.0.smithi169.stderr:+ fio --name=fiotest --filename=/dev/rbd14 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-17T00:43:51.797 INFO:tasks.workunit.client.0.smithi169.stderr:+ DEVS[$i]=/dev/rbd15 2023-10-17T00:43:51.797 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:51.798 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:51.798 INFO:tasks.workunit.client.0.smithi169.stderr:+ fio --name=fiotest --filename=/dev/rbd15 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-17T00:43:51.804 INFO:tasks.workunit.client.0.smithi169.stderr:++ sudo rbd device map rbd/image17 2023-10-17T00:43:51.975 INFO:tasks.workunit.client.0.smithi169.stderr:+ DEVS[$i]=/dev/rbd16 2023-10-17T00:43:51.975 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:51.975 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:51.976 INFO:tasks.workunit.client.0.smithi169.stderr:+ fio --name=fiotest --filename=/dev/rbd16 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-17T00:43:51.976 INFO:tasks.workunit.client.0.smithi169.stderr:++ sudo rbd device map rbd/image18 2023-10-17T00:43:52.118 INFO:tasks.workunit.client.0.smithi169.stderr:+ DEVS[$i]=/dev/rbd17 2023-10-17T00:43:52.118 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:52.118 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:52.129 INFO:tasks.workunit.client.0.smithi169.stderr:+ fio --name=fiotest --filename=/dev/rbd17 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-17T00:43:52.129 INFO:tasks.workunit.client.0.smithi169.stderr:++ sudo rbd device map rbd/image19 2023-10-17T00:43:52.282 INFO:tasks.workunit.client.0.smithi169.stderr:+ DEVS[$i]=/dev/rbd18 2023-10-17T00:43:52.283 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:52.283 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:52.284 INFO:tasks.workunit.client.0.smithi169.stderr:+ fio --name=fiotest --filename=/dev/rbd18 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-17T00:43:52.286 INFO:tasks.workunit.client.0.smithi169.stderr:++ sudo rbd device map rbd/image20 2023-10-17T00:43:52.434 INFO:tasks.workunit.client.0.smithi169.stderr:+ DEVS[$i]=/dev/rbd19 2023-10-17T00:43:52.435 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T00:43:52.435 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 20 )) 2023-10-17T00:43:52.435 INFO:tasks.workunit.client.0.smithi169.stderr:+ fio --name=fiotest --filename=/dev/rbd19 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-17T00:43:52.437 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:43:52.439 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503432 2023-10-17T00:43:52.440 INFO:tasks.workunit.client.0.smithi169.stderr:+ END_TIME=1697507032 2023-10-17T00:43:52.440 INFO:tasks.workunit.client.0.smithi169.stderr:+ PREV_CLIENT_ADDR= 2023-10-17T00:43:52.440 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR= 2023-10-17T00:43:52.440 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:43:52.440 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n '' ]] 2023-10-17T00:43:52.441 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:44:02.442 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:44:02.442 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:44:02.443 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:44:02.443 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:44:03.438 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/3228280117 2023-10-17T00:44:03.438 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:44:03.441 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503443 2023-10-17T00:44:03.441 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:44:03.441 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/3228280117 ]] 2023-10-17T00:44:03.441 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/3228280117 != '' ]] 2023-10-17T00:44:03.442 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist add 172.21.15.169:0/3228280117 2023-10-17T00:44:05.759 INFO:tasks.workunit.client.0.smithi169.stderr:blocklisting 172.21.15.169:0/3228280117 until 2023-10-17T01:44:04.992720+0000 (3600 sec) 2023-10-17T00:44:05.796 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist ls 2023-10-17T00:44:05.797 INFO:tasks.workunit.client.0.smithi169.stderr:+ grep -q 172.21.15.169:0/3228280117 2023-10-17T00:44:06.513 INFO:tasks.workunit.client.0.smithi169.stderr:listed 1 entries 2023-10-17T00:44:06.555 INFO:tasks.workunit.client.0.smithi169.stderr:+ PREV_CLIENT_ADDR=172.21.15.169:0/3228280117 2023-10-17T00:44:06.555 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:44:16.557 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:44:16.557 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:44:16.558 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:44:16.576 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:44:17.433 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/3228280117 2023-10-17T00:44:17.434 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:44:17.435 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503457 2023-10-17T00:44:17.436 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:44:17.436 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/3228280117 ]] 2023-10-17T00:44:17.436 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/3228280117 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\3\2\2\8\2\8\0\1\1\7 ]] 2023-10-17T00:44:17.436 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:44:27.438 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:44:27.439 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:44:27.441 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:44:27.444 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:44:28.369 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/3228280117 2023-10-17T00:44:28.369 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:44:28.370 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503468 2023-10-17T00:44:28.370 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:44:28.371 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/3228280117 ]] 2023-10-17T00:44:28.371 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/3228280117 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\3\2\2\8\2\8\0\1\1\7 ]] 2023-10-17T00:44:28.371 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:44:38.383 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:44:38.383 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:44:38.384 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:44:38.384 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:44:39.258 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/3228280117 2023-10-17T00:44:39.261 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:44:39.262 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503479 2023-10-17T00:44:39.263 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:44:39.263 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/3228280117 ]] 2023-10-17T00:44:39.263 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/3228280117 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\3\2\2\8\2\8\0\1\1\7 ]] 2023-10-17T00:44:39.263 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:44:39.849 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:44:39.842+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T00:44:39.850 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:44:39.846+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T00:44:39.850 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:44:39.846+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbda750 2023-10-17T00:44:39.850 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:44:39.846+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T00:44:39.850 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:44:39.846+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T00:44:39.850 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:44:39.846+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T00:44:39.851 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:44:39.846+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T00:44:39.851 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:44:39.846+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T00:44:39.851 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:44:39.846+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T00:44:39.851 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:44:39.846+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T00:44:39.851 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:44:39.846+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T00:44:39.852 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:44:39.846+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T00:44:39.852 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:44:39.846+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T00:44:39.852 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:44:39.846+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T00:44:39.852 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:44:39.846+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T00:44:49.264 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:44:49.265 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:44:49.265 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:44:49.279 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:44:50.184 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/3228280117 2023-10-17T00:44:50.185 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:44:50.185 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503490 2023-10-17T00:44:50.186 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:44:50.186 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/3228280117 ]] 2023-10-17T00:44:50.186 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/3228280117 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\3\2\2\8\2\8\0\1\1\7 ]] 2023-10-17T00:44:50.186 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:45:00.004 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.002+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.005 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.002+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b3c8d780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.006 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.006+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.006 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.006+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4385b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.011 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.010+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.012 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.010+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4384100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.015 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.014+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.015 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.014+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4385f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.018 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.018+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.019 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.018+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b3c8d700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.021 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.022+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.021 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.022+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b3c8c780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.023 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.022+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.024 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.022+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4384980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.027 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.026+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.027 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.026+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4384a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.030 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.030+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.031 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.030+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b3c8d500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.033 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.034+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.033 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.034+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b3c8d200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.036 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.034+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.037 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.034+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b3c8d580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.039 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.038+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.039 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.038+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b3abc900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.042 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.042+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.042 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.042+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4385f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.045 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.046+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.046 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.046+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4385b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.049 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.046+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.049 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.046+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4384580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.051 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.050+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.051 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.050+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b3c8c080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.055 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.054+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.055 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.054+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b3c8c500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.057 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.058+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.058 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.058+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4384100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.061 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.062+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.061 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.062+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4384f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.066 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.062+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.066 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:45:00.062+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b3c8d600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:45:00.189 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:45:00.191 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:45:00.196 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:45:00.208 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:45:01.285 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/3228280117 2023-10-17T00:45:01.292 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:45:01.294 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503501 2023-10-17T00:45:01.294 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:45:01.294 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/3228280117 ]] 2023-10-17T00:45:01.294 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/3228280117 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\3\2\2\8\2\8\0\1\1\7 ]] 2023-10-17T00:45:01.295 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:45:11.298 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:45:11.303 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:45:11.304 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:45:11.305 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:45:12.162 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/3228280117 2023-10-17T00:45:12.164 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:45:12.165 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503512 2023-10-17T00:45:12.166 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:45:12.166 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/3228280117 ]] 2023-10-17T00:45:12.166 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/3228280117 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\3\2\2\8\2\8\0\1\1\7 ]] 2023-10-17T00:45:12.167 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:45:22.169 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:45:22.170 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:45:22.170 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:45:22.176 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:45:23.091 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1553406650 2023-10-17T00:45:23.091 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:45:23.092 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503523 2023-10-17T00:45:23.092 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:45:23.092 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1553406650 ]] 2023-10-17T00:45:23.093 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1553406650 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\3\2\2\8\2\8\0\1\1\7 ]] 2023-10-17T00:45:23.093 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist add 172.21.15.169:0/1553406650 2023-10-17T00:45:24.798 INFO:tasks.workunit.client.0.smithi169.stderr:blocklisting 172.21.15.169:0/1553406650 until 2023-10-17T01:45:24.006091+0000 (3600 sec) 2023-10-17T00:45:24.845 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist ls 2023-10-17T00:45:24.846 INFO:tasks.workunit.client.0.smithi169.stderr:+ grep -q 172.21.15.169:0/1553406650 2023-10-17T00:45:25.516 INFO:tasks.workunit.client.0.smithi169.stderr:listed 2 entries 2023-10-17T00:45:25.549 INFO:tasks.workunit.client.0.smithi169.stderr:+ PREV_CLIENT_ADDR=172.21.15.169:0/1553406650 2023-10-17T00:45:25.549 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:45:35.551 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:45:35.551 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:45:35.554 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:45:35.556 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:45:36.403 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1553406650 2023-10-17T00:45:36.408 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:45:36.409 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503536 2023-10-17T00:45:36.409 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:45:36.409 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1553406650 ]] 2023-10-17T00:45:36.409 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1553406650 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\5\5\3\4\0\6\6\5\0 ]] 2023-10-17T00:45:36.410 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:45:46.410 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:45:46.411 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:45:46.411 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:45:46.415 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:45:47.302 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1553406650 2023-10-17T00:45:47.302 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:45:47.303 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503547 2023-10-17T00:45:47.303 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:45:47.303 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1553406650 ]] 2023-10-17T00:45:47.303 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1553406650 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\5\5\3\4\0\6\6\5\0 ]] 2023-10-17T00:45:47.304 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:45:56.894 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:45:56.894+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T00:45:56.895 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:45:56.894+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T00:45:57.305 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:45:57.305 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:45:57.312 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:45:57.317 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:45:58.153 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1553406650 2023-10-17T00:45:58.153 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:45:58.154 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503558 2023-10-17T00:45:58.154 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:45:58.154 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1553406650 ]] 2023-10-17T00:45:58.154 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1553406650 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\5\5\3\4\0\6\6\5\0 ]] 2023-10-17T00:45:58.155 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:46:00.002 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.001+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.003 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.001+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b3abdc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.007 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.005+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.007 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.005+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b46a5000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.012 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.013+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.013 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.013+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b46a5200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.018 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.017+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.018 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.017+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b46a5400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.022 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.021+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.023 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.021+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b46a5600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.028 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.029+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.029 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.029+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b3abdc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.033 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.033+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.033 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.033+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b46a5480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.035 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.033+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.036 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.033+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4385380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.039 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.037+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.039 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.037+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b46a4f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.046 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.045+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.046 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.045+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b46a5000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.050 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.049+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.050 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.049+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b3abda00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.054 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.053+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.054 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.053+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b3af4b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.058 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.057+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.058 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.057+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b46a5100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.063 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.061+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.063 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.061+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b46a5300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.066 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.065+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.066 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.065+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b46a5480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.069 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.069+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.069 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.069+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b46a4f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.073 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.073+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.073 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.073+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b46a5200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.084 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.085+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.084 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.085+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b3abda00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.089 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.089+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.089 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.089+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b431f080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.095 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.093+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:00.095 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:46:00.093+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b3abc800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:46:08.164 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:46:08.165 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:46:08.180 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:46:08.198 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:46:09.190 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1553406650 2023-10-17T00:46:09.190 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:46:09.191 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503569 2023-10-17T00:46:09.191 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:46:09.191 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1553406650 ]] 2023-10-17T00:46:09.191 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1553406650 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\5\5\3\4\0\6\6\5\0 ]] 2023-10-17T00:46:09.192 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:46:19.192 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:46:19.193 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:46:19.193 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:46:19.212 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:46:20.163 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1553406650 2023-10-17T00:46:20.163 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:46:20.164 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503580 2023-10-17T00:46:20.164 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:46:20.164 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1553406650 ]] 2023-10-17T00:46:20.164 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1553406650 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\5\5\3\4\0\6\6\5\0 ]] 2023-10-17T00:46:20.165 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:46:30.168 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:46:30.169 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:46:30.172 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:46:30.184 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:46:31.258 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/118575510 2023-10-17T00:46:31.258 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:46:31.259 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503591 2023-10-17T00:46:31.259 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:46:31.259 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/118575510 ]] 2023-10-17T00:46:31.259 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/118575510 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\5\5\3\4\0\6\6\5\0 ]] 2023-10-17T00:46:31.259 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist add 172.21.15.169:0/118575510 2023-10-17T00:46:33.784 INFO:tasks.workunit.client.0.smithi169.stderr:blocklisting 172.21.15.169:0/118575510 until 2023-10-17T01:46:33.026037+0000 (3600 sec) 2023-10-17T00:46:33.824 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist ls 2023-10-17T00:46:33.824 INFO:tasks.workunit.client.0.smithi169.stderr:+ grep -q 172.21.15.169:0/118575510 2023-10-17T00:46:34.587 INFO:tasks.workunit.client.0.smithi169.stderr:listed 3 entries 2023-10-17T00:46:34.610 INFO:tasks.workunit.client.0.smithi169.stderr:+ PREV_CLIENT_ADDR=172.21.15.169:0/118575510 2023-10-17T00:46:34.610 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:46:44.614 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:46:44.614 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:46:44.615 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:46:44.632 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:46:45.613 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/118575510 2023-10-17T00:46:45.613 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:46:45.614 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503605 2023-10-17T00:46:45.614 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:46:45.614 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/118575510 ]] 2023-10-17T00:46:45.615 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/118575510 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\1\8\5\7\5\5\1\0 ]] 2023-10-17T00:46:45.615 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:46:55.616 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:46:55.616 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:46:55.618 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:46:55.635 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:46:56.579 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/118575510 2023-10-17T00:46:56.580 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:46:56.580 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503616 2023-10-17T00:46:56.581 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:46:56.581 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/118575510 ]] 2023-10-17T00:46:56.581 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/118575510 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\1\8\5\7\5\5\1\0 ]] 2023-10-17T00:46:56.581 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:47:00.001 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.001+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.002 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.001+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b3b0f400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.006 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.005+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.006 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.005+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b3b0fb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.011 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.009+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.011 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.009+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b3b0fd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.017 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.017+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.018 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.017+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b3b0ff80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.022 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.021+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.023 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.021+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4928180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.034 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.033+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.034 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.033+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b46a4e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.045 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.045+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.045 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.045+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b46a5300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.049 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.049+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.049 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.049+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b46a5180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.058 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.057+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.058 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.057+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.059 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.057+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b3b0fe00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.059 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.057+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b3b0fb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.061 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.061+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.061 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.061+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b3b0f400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.065 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.065+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.065 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.065+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b46a4480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.069 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.069+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.069 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.069+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4928400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.072 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.073+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.072 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.073+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4928600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.081 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.081+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.082 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.081+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b3b0fc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.084 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.085+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.084 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.085+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4928800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.090 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.089+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.091 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.089+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4928a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.098 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.097+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.098 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.097+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4928c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.101 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.101+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.101 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.101+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4929600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.109 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.109+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:00.110 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:47:00.109+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4929480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:47:06.582 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:47:06.582 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:47:06.583 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:47:06.600 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:47:07.574 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/118575510 2023-10-17T00:47:07.575 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:47:07.576 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503627 2023-10-17T00:47:07.576 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:47:07.577 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/118575510 ]] 2023-10-17T00:47:07.577 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/118575510 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\1\8\5\7\5\5\1\0 ]] 2023-10-17T00:47:07.577 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:47:08.696 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:47:08.693+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T00:47:08.696 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:47:08.693+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T00:47:08.696 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:47:08.693+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T00:47:08.697 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:47:08.693+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T00:47:08.697 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:47:08.693+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T00:47:08.697 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:47:08.693+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T00:47:08.697 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:47:08.693+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T00:47:08.697 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:47:08.693+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T00:47:08.698 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:47:08.693+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T00:47:08.698 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:47:08.693+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T00:47:08.698 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:47:08.693+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T00:47:08.698 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:47:08.693+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T00:47:17.579 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:47:17.580 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:47:17.583 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:47:17.596 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:47:18.430 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/118575510 2023-10-17T00:47:18.432 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:47:18.433 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503638 2023-10-17T00:47:18.434 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:47:18.434 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/118575510 ]] 2023-10-17T00:47:18.434 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/118575510 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\1\8\5\7\5\5\1\0 ]] 2023-10-17T00:47:18.434 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:47:28.440 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:47:28.440 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:47:28.442 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:47:28.454 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:47:29.318 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/118575510 2023-10-17T00:47:29.319 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:47:29.319 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503649 2023-10-17T00:47:29.319 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:47:29.319 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/118575510 ]] 2023-10-17T00:47:29.319 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/118575510 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\1\8\5\7\5\5\1\0 ]] 2023-10-17T00:47:29.320 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:47:39.319 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:47:39.328 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:47:39.335 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:47:39.337 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:47:40.214 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/4110034747 2023-10-17T00:47:40.214 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:47:40.214 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503660 2023-10-17T00:47:40.215 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:47:40.215 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/4110034747 ]] 2023-10-17T00:47:40.215 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/4110034747 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\1\8\5\7\5\5\1\0 ]] 2023-10-17T00:47:40.215 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist add 172.21.15.169:0/4110034747 2023-10-17T00:47:41.797 INFO:tasks.workunit.client.0.smithi169.stderr:blocklisting 172.21.15.169:0/4110034747 until 2023-10-17T01:47:40.992687+0000 (3600 sec) 2023-10-17T00:47:41.840 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist ls 2023-10-17T00:47:41.843 INFO:tasks.workunit.client.0.smithi169.stderr:+ grep -q 172.21.15.169:0/4110034747 2023-10-17T00:47:42.565 INFO:tasks.workunit.client.0.smithi169.stderr:listed 4 entries 2023-10-17T00:47:42.610 INFO:tasks.workunit.client.0.smithi169.stderr:+ PREV_CLIENT_ADDR=172.21.15.169:0/4110034747 2023-10-17T00:47:42.610 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:47:52.621 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:47:52.621 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:47:52.644 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:47:52.646 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:47:53.587 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/4110034747 2023-10-17T00:47:53.588 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:47:53.588 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503673 2023-10-17T00:47:53.589 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:47:53.589 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/4110034747 ]] 2023-10-17T00:47:53.589 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/4110034747 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\4\1\1\0\0\3\4\7\4\7 ]] 2023-10-17T00:47:53.589 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:48:00.004 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.004+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.004 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.004+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b3b0ea00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.005 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.004+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.005 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.004+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b49c5380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.009 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.008+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.009 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.008+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4a3e380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.014 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.012+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.014 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.012+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b49c4600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.023 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.020+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.023 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.020+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4a3e480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.025 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.024+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.025 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.024+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4a3ef80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.034 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.032+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.034 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.032+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4a3f580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.038 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.036+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.039 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.036+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4a3f400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.051 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.052+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.052 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.052+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4a3fb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.056 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.056+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.056 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.056+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4a3f980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.061 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.060+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.062 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.060+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4a3f780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.077 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.076+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.077 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.076+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4a3f200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.081 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.080+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.082 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.080+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4a3fd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.086 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.084+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.086 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.084+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4a3ff80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.106 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.104+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.106 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.104+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4a3e500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.111 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.112+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.112 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.112+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4a3e680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.116 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.116+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.116 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.116+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4a3f080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.126 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.124+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.126 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.124+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4a3f500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.131 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.132+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.131 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.132+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4a3f300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.136 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.136+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:00.136 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:00.136+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4a3fa80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:48:03.590 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:48:03.590 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:48:03.591 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:48:03.591 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:48:04.484 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/4110034747 2023-10-17T00:48:04.497 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:48:04.497 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503684 2023-10-17T00:48:04.498 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:48:04.498 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/4110034747 ]] 2023-10-17T00:48:04.498 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/4110034747 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\4\1\1\0\0\3\4\7\4\7 ]] 2023-10-17T00:48:04.498 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:48:14.500 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:48:14.504 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:48:14.505 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:48:14.522 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:48:15.413 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/4110034747 2023-10-17T00:48:15.414 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:48:15.414 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503695 2023-10-17T00:48:15.415 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:48:15.415 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/4110034747 ]] 2023-10-17T00:48:15.415 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/4110034747 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\4\1\1\0\0\3\4\7\4\7 ]] 2023-10-17T00:48:15.415 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:48:25.416 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:48:25.417 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:48:25.418 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:48:25.436 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:48:26.275 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/4110034747 2023-10-17T00:48:26.275 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:48:26.276 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503706 2023-10-17T00:48:26.276 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:48:26.276 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/4110034747 ]] 2023-10-17T00:48:26.276 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/4110034747 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\4\1\1\0\0\3\4\7\4\7 ]] 2023-10-17T00:48:26.276 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:48:36.279 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:48:36.279 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:48:36.281 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:48:36.287 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:48:37.174 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/4110034747 2023-10-17T00:48:37.175 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:48:37.175 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503717 2023-10-17T00:48:37.176 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:48:37.176 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/4110034747 ]] 2023-10-17T00:48:37.176 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/4110034747 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\4\1\1\0\0\3\4\7\4\7 ]] 2023-10-17T00:48:37.176 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:48:47.183 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:48:47.183 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:48:47.187 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:48:47.202 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:48:48.095 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/2848467669 2023-10-17T00:48:48.099 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:48:48.100 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503728 2023-10-17T00:48:48.100 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:48:48.100 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/2848467669 ]] 2023-10-17T00:48:48.101 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/2848467669 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\4\1\1\0\0\3\4\7\4\7 ]] 2023-10-17T00:48:48.101 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist add 172.21.15.169:0/2848467669 2023-10-17T00:48:49.833 INFO:tasks.workunit.client.0.smithi169.stderr:blocklisting 172.21.15.169:0/2848467669 until 2023-10-17T01:48:49.052709+0000 (3600 sec) 2023-10-17T00:48:49.880 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist ls 2023-10-17T00:48:49.880 INFO:tasks.workunit.client.0.smithi169.stderr:+ grep -q 172.21.15.169:0/2848467669 2023-10-17T00:48:50.664 INFO:tasks.workunit.client.0.smithi169.stderr:listed 5 entries 2023-10-17T00:48:50.706 INFO:tasks.workunit.client.0.smithi169.stderr:+ PREV_CLIENT_ADDR=172.21.15.169:0/2848467669 2023-10-17T00:48:50.706 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:49:00.000 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:59.999+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.001 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:48:59.999+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b49c4000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.004 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.003+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.004 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.003+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4acd000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.008 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.007+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.008 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.007+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4acc680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.017 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.015+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.017 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.015+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4acc580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.022 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.019+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.022 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.019+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4acc380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.030 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.027+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.030 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.027+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4acd300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.033 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.031+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.033 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.031+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4acd900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.042 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.039+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.043 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.039+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4acd780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.049 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.043+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.049 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.047+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4acde80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.056 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.055+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.056 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.055+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4acdd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.059 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.059+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.060 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.059+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4acdb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.068 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.067+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.068 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.067+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4acd580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.070 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.071+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.071 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.071+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4bcc100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.073 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.071+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.073 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.071+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4bcc300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.076 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.075+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.076 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.075+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4bcc500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.081 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.079+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.082 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.079+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4bcc700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.083 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.079+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.083 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.079+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4bcc900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.091 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.091+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.091 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.091+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4bccb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.101 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.099+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.101 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.099+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4bccd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.110 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.107+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.110 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:49:00.107+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4bccf00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:49:00.711 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:49:00.711 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:49:00.712 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:49:00.712 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:49:01.539 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/2848467669 2023-10-17T00:49:01.540 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:49:01.540 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503741 2023-10-17T00:49:01.540 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:49:01.541 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/2848467669 ]] 2023-10-17T00:49:01.541 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/2848467669 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\2\8\4\8\4\6\7\6\6\9 ]] 2023-10-17T00:49:01.541 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:49:11.547 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:49:11.548 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:49:11.548 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:49:11.549 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:49:12.560 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/2848467669 2023-10-17T00:49:12.561 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:49:12.562 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503752 2023-10-17T00:49:12.563 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:49:12.563 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/2848467669 ]] 2023-10-17T00:49:12.563 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/2848467669 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\2\8\4\8\4\6\7\6\6\9 ]] 2023-10-17T00:49:12.563 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:49:22.564 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:49:22.565 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:49:22.567 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:49:22.580 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:49:23.605 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/2848467669 2023-10-17T00:49:23.606 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:49:23.608 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503763 2023-10-17T00:49:23.608 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:49:23.608 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/2848467669 ]] 2023-10-17T00:49:23.608 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/2848467669 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\2\8\4\8\4\6\7\6\6\9 ]] 2023-10-17T00:49:23.608 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:49:33.611 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:49:33.611 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:49:33.612 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:49:33.618 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:49:34.622 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/2848467669 2023-10-17T00:49:34.632 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:49:34.633 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503774 2023-10-17T00:49:34.633 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:49:34.633 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/2848467669 ]] 2023-10-17T00:49:34.633 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/2848467669 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\2\8\4\8\4\6\7\6\6\9 ]] 2023-10-17T00:49:34.634 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:49:44.640 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:49:44.641 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:49:44.641 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:49:44.642 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:49:45.574 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/2848467669 2023-10-17T00:49:45.575 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:49:45.576 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503785 2023-10-17T00:49:45.576 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:49:45.576 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/2848467669 ]] 2023-10-17T00:49:45.576 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/2848467669 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\2\8\4\8\4\6\7\6\6\9 ]] 2023-10-17T00:49:45.577 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:49:55.587 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:49:55.590 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:49:55.595 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:49:55.605 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:49:56.462 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/3869173669 2023-10-17T00:49:56.467 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:49:56.468 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503796 2023-10-17T00:49:56.468 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:49:56.469 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/3869173669 ]] 2023-10-17T00:49:56.469 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/3869173669 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\2\8\4\8\4\6\7\6\6\9 ]] 2023-10-17T00:49:56.469 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist add 172.21.15.169:0/3869173669 2023-10-17T00:49:58.874 INFO:tasks.workunit.client.0.smithi169.stderr:blocklisting 172.21.15.169:0/3869173669 until 2023-10-17T01:49:58.104004+0000 (3600 sec) 2023-10-17T00:49:58.929 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist ls 2023-10-17T00:49:58.930 INFO:tasks.workunit.client.0.smithi169.stderr:+ grep -q 172.21.15.169:0/3869173669 2023-10-17T00:49:59.757 INFO:tasks.workunit.client.0.smithi169.stderr:listed 6 entries 2023-10-17T00:49:59.792 INFO:tasks.workunit.client.0.smithi169.stderr:+ PREV_CLIENT_ADDR=172.21.15.169:0/3869173669 2023-10-17T00:49:59.792 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:50:00.000 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.002+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.000 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.002+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4385f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.003 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.006+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.003 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.006+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b3b0f180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.016 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.018+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.017 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.018+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b3c8d900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.021 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.022+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.021 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.022+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b3b0f280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.028 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.030+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.028 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.030+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b3c8dc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.032 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.034+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.033 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.034+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4929180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.036 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.038+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.036 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.038+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4928f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.042 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.042+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.042 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.042+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4929d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.048 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.050+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.048 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.050+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4a3f780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.055 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.058+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.055 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.058+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4a3ec00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.059 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.062+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.059 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.062+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4928e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.065 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.066+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.065 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.066+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4929300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.070 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.070+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.070 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.070+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4a3e480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.075 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.078+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.076 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.078+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4a3e680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.080 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.082+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.080 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.082+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4a3ef80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.085 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.086+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.085 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.086+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b49c4200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.090 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.090+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.090 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.090+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4a3fc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.094 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.094+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.094 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.094+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4a3f600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.098 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.098+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.099 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.098+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b49c5700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.103 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.106+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:00.103 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:50:00.106+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b49c5000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:50:03.945 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:50:03.942+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T00:50:03.945 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:50:03.942+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T00:50:03.946 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:50:03.946+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T00:50:03.946 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:50:03.946+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T00:50:03.946 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:50:03.946+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T00:50:03.947 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:50:03.946+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T00:50:09.800 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:50:09.801 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:50:09.809 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:50:09.810 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:50:10.750 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/3869173669 2023-10-17T00:50:10.750 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:50:10.751 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503810 2023-10-17T00:50:10.751 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:50:10.751 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/3869173669 ]] 2023-10-17T00:50:10.752 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/3869173669 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\3\8\6\9\1\7\3\6\6\9 ]] 2023-10-17T00:50:10.752 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:50:20.758 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:50:20.759 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:50:20.759 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:50:20.779 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:50:21.555 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/3869173669 2023-10-17T00:50:21.556 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:50:21.557 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503821 2023-10-17T00:50:21.557 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:50:21.557 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/3869173669 ]] 2023-10-17T00:50:21.557 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/3869173669 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\3\8\6\9\1\7\3\6\6\9 ]] 2023-10-17T00:50:21.557 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:50:31.558 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:50:31.573 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:50:31.578 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:50:31.580 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:50:32.593 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/3869173669 2023-10-17T00:50:32.594 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:50:32.594 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503832 2023-10-17T00:50:32.595 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:50:32.595 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/3869173669 ]] 2023-10-17T00:50:32.595 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/3869173669 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\3\8\6\9\1\7\3\6\6\9 ]] 2023-10-17T00:50:32.595 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:50:42.603 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:50:42.603 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:50:42.609 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:50:42.610 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:50:43.513 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/3869173669 2023-10-17T00:50:43.518 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:50:43.530 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503843 2023-10-17T00:50:43.530 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:50:43.531 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/3869173669 ]] 2023-10-17T00:50:43.531 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/3869173669 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\3\8\6\9\1\7\3\6\6\9 ]] 2023-10-17T00:50:43.531 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:50:53.548 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:50:53.549 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:50:53.549 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:50:53.549 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:50:54.505 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/3869173669 2023-10-17T00:50:54.506 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:50:54.506 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503854 2023-10-17T00:50:54.506 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:50:54.507 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/3869173669 ]] 2023-10-17T00:50:54.507 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/3869173669 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\3\8\6\9\1\7\3\6\6\9 ]] 2023-10-17T00:50:54.507 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:51:04.510 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:51:04.510 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:51:04.511 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:51:04.513 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:51:05.338 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1704238022 2023-10-17T00:51:05.338 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:51:05.348 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503865 2023-10-17T00:51:05.348 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:51:05.348 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1704238022 ]] 2023-10-17T00:51:05.348 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1704238022 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\3\8\6\9\1\7\3\6\6\9 ]] 2023-10-17T00:51:05.349 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist add 172.21.15.169:0/1704238022 2023-10-17T00:51:06.875 INFO:tasks.workunit.client.0.smithi169.stderr:blocklisting 172.21.15.169:0/1704238022 until 2023-10-17T01:51:06.068633+0000 (3600 sec) 2023-10-17T00:51:06.903 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist ls 2023-10-17T00:51:06.904 INFO:tasks.workunit.client.0.smithi169.stderr:+ grep -q 172.21.15.169:0/1704238022 2023-10-17T00:51:07.725 INFO:tasks.workunit.client.0.smithi169.stderr:listed 7 entries 2023-10-17T00:51:07.756 INFO:tasks.workunit.client.0.smithi169.stderr:+ PREV_CLIENT_ADDR=172.21.15.169:0/1704238022 2023-10-17T00:51:07.756 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:51:17.757 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:51:17.758 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:51:17.758 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:51:17.764 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:51:18.753 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1704238022 2023-10-17T00:51:18.754 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:51:18.759 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503878 2023-10-17T00:51:18.759 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:51:18.759 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1704238022 ]] 2023-10-17T00:51:18.759 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1704238022 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\7\0\4\2\3\8\0\2\2 ]] 2023-10-17T00:51:18.759 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:51:28.760 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:51:28.762 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:51:28.764 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:51:28.775 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:51:29.593 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1704238022 2023-10-17T00:51:29.594 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:51:29.596 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503889 2023-10-17T00:51:29.596 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:51:29.596 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1704238022 ]] 2023-10-17T00:51:29.597 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1704238022 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\7\0\4\2\3\8\0\2\2 ]] 2023-10-17T00:51:29.597 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:51:39.597 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:51:39.599 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:51:39.600 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:51:39.613 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:51:40.515 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1704238022 2023-10-17T00:51:40.515 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:51:40.516 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503900 2023-10-17T00:51:40.516 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:51:40.516 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1704238022 ]] 2023-10-17T00:51:40.516 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1704238022 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\7\0\4\2\3\8\0\2\2 ]] 2023-10-17T00:51:40.517 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:51:50.529 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:51:50.541 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:51:50.542 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:51:50.542 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:51:51.449 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1704238022 2023-10-17T00:51:51.450 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:51:51.451 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503911 2023-10-17T00:51:51.451 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:51:51.452 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1704238022 ]] 2023-10-17T00:51:51.452 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1704238022 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\7\0\4\2\3\8\0\2\2 ]] 2023-10-17T00:51:51.452 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:52:00.002 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.000+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.003 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.000+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b431e780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.003 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.004+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.003 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.004+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b431ef00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.009 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.008+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.009 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.008+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b3abce00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.015 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.016+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.015 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.016+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b431e900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.019 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.020+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.019 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.020+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4bcdc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.033 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.032+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.034 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.032+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4bcdb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.045 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.044+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.045 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.044+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b46a4f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.046 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.044+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.046 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.044+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4bcd600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.062 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.060+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.062 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.060+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4a3ea00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.070 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.072+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.071 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.072+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4bcd880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.074 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.072+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.074 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.072+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4385080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.078 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.076+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.078 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.076+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b46a4b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.088 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.088+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.088 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.088+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b49c5b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.099 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.100+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.099 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.100+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4a3ed80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.109 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.108+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.109 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.108+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4a3eb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.113 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.112+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.113 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.112+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4acd300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.116 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.116+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.116 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.116+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4acdc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.119 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.120+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.119 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.120+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b49c4700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.129 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.128+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.129 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.128+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4ad5300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.132 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.132+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:00.132 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:00.132+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4ad5180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:52:01.465 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:52:01.465 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:52:01.466 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:52:01.466 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:52:02.382 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1704238022 2023-10-17T00:52:02.382 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:52:02.383 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503922 2023-10-17T00:52:02.384 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:52:02.384 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1704238022 ]] 2023-10-17T00:52:02.384 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1704238022 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\7\0\4\2\3\8\0\2\2 ]] 2023-10-17T00:52:02.384 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:52:12.387 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:52:12.390 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:52:12.391 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:52:12.391 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:52:13.224 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1704238022 2023-10-17T00:52:13.224 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:52:13.225 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503933 2023-10-17T00:52:13.225 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:52:13.225 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1704238022 ]] 2023-10-17T00:52:13.225 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1704238022 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\7\0\4\2\3\8\0\2\2 ]] 2023-10-17T00:52:13.226 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:52:23.226 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:52:23.228 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:52:23.229 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:52:23.247 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:52:24.028 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/895333782 2023-10-17T00:52:24.028 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:52:24.029 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503944 2023-10-17T00:52:24.029 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:52:24.029 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/895333782 ]] 2023-10-17T00:52:24.029 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/895333782 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\7\0\4\2\3\8\0\2\2 ]] 2023-10-17T00:52:24.030 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist add 172.21.15.169:0/895333782 2023-10-17T00:52:25.915 INFO:tasks.workunit.client.0.smithi169.stderr:blocklisting 172.21.15.169:0/895333782 until 2023-10-17T01:52:25.116678+0000 (3600 sec) 2023-10-17T00:52:25.969 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist ls 2023-10-17T00:52:25.969 INFO:tasks.workunit.client.0.smithi169.stderr:+ grep -q 172.21.15.169:0/895333782 2023-10-17T00:52:26.660 INFO:tasks.workunit.client.0.smithi169.stderr:listed 8 entries 2023-10-17T00:52:26.708 INFO:tasks.workunit.client.0.smithi169.stderr:+ PREV_CLIENT_ADDR=172.21.15.169:0/895333782 2023-10-17T00:52:26.708 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:52:36.711 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:52:36.713 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:52:36.714 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:52:36.717 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:52:37.579 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/895333782 2023-10-17T00:52:37.579 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:52:37.580 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503957 2023-10-17T00:52:37.580 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:52:37.580 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/895333782 ]] 2023-10-17T00:52:37.580 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/895333782 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\8\9\5\3\3\3\7\8\2 ]] 2023-10-17T00:52:37.581 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:52:47.581 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:52:47.582 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:52:47.582 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:52:47.585 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:52:48.548 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/895333782 2023-10-17T00:52:48.548 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:52:48.555 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503968 2023-10-17T00:52:48.555 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:52:48.556 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/895333782 ]] 2023-10-17T00:52:48.556 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/895333782 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\8\9\5\3\3\3\7\8\2 ]] 2023-10-17T00:52:48.556 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:52:58.558 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:52:58.558 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:52:58.560 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:52:58.560 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:52:59.464 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/895333782 2023-10-17T00:52:59.466 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:52:59.467 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503979 2023-10-17T00:52:59.467 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:52:59.468 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/895333782 ]] 2023-10-17T00:52:59.468 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/895333782 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\8\9\5\3\3\3\7\8\2 ]] 2023-10-17T00:52:59.468 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:53:00.002 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:59.999+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.002 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:52:59.999+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4ad5a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.002 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.003+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.003 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.003+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4ad5c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.005 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.003+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.005 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.003+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4ad5e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.018 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.019+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.018 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.019+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4d32000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.021 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.019+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.021 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.019+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4d32580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.025 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.023+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.025 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.023+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4d32400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.029 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.027+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.029 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.027+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4d32280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.032 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.031+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.033 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.031+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4d32800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.035 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.035+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.035 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.035+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4d32a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.046 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.047+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.046 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.047+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4d32c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.050 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.051+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.050 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.051+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4d32e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.054 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.055+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.054 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.055+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4d33600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.060 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.059+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.061 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.059+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4d33480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.065 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.063+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.065 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.063+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4d33280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.068 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.067+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.069 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.067+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4d33d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.074 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.075+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.074 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.075+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4d33c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.078 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.079+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.078 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.079+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4d33a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.082 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.083+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.082 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.083+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4d33800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.086 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.087+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.087 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.087+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4d33080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.091 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.091+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:00.091 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:53:00.091+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4408000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:53:09.471 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:53:09.471 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:53:09.472 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:53:09.477 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:53:10.513 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/895333782 2023-10-17T00:53:10.514 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:53:10.521 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697503990 2023-10-17T00:53:10.521 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:53:10.521 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/895333782 ]] 2023-10-17T00:53:10.521 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/895333782 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\8\9\5\3\3\3\7\8\2 ]] 2023-10-17T00:53:10.522 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:53:20.392 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:53:20.387+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T00:53:20.394 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:53:20.387+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T00:53:20.394 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:53:20.387+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbda750 2023-10-17T00:53:20.394 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:53:20.387+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T00:53:20.394 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:53:20.387+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T00:53:20.394 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:53:20.387+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T00:53:20.395 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:53:20.387+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T00:53:20.395 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:53:20.387+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T00:53:20.395 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:53:20.387+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T00:53:20.395 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:53:20.387+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T00:53:20.395 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:53:20.387+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T00:53:20.396 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:53:20.387+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T00:53:20.396 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:53:20.387+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T00:53:20.396 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:53:20.387+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T00:53:20.396 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:53:20.387+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T00:53:20.523 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:53:20.523 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:53:20.533 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:53:20.535 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:53:21.444 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/895333782 2023-10-17T00:53:21.445 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:53:21.445 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504001 2023-10-17T00:53:21.445 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:53:21.446 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/895333782 ]] 2023-10-17T00:53:21.446 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/895333782 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\8\9\5\3\3\3\7\8\2 ]] 2023-10-17T00:53:21.446 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:53:31.230 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:53:31.227+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T00:53:31.234 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:53:31.231+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T00:53:31.235 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:53:31.231+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T00:53:31.456 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:53:31.460 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:53:31.470 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:53:31.472 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:53:32.341 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/715849107 2023-10-17T00:53:32.341 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:53:32.342 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504012 2023-10-17T00:53:32.342 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:53:32.343 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/715849107 ]] 2023-10-17T00:53:32.343 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/715849107 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\8\9\5\3\3\3\7\8\2 ]] 2023-10-17T00:53:32.343 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist add 172.21.15.169:0/715849107 2023-10-17T00:53:33.900 INFO:tasks.workunit.client.0.smithi169.stderr:blocklisting 172.21.15.169:0/715849107 until 2023-10-17T01:53:33.087303+0000 (3600 sec) 2023-10-17T00:53:33.978 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist ls 2023-10-17T00:53:33.978 INFO:tasks.workunit.client.0.smithi169.stderr:+ grep -q 172.21.15.169:0/715849107 2023-10-17T00:53:34.735 INFO:tasks.workunit.client.0.smithi169.stderr:listed 9 entries 2023-10-17T00:53:34.762 INFO:tasks.workunit.client.0.smithi169.stderr:+ PREV_CLIENT_ADDR=172.21.15.169:0/715849107 2023-10-17T00:53:34.763 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:53:44.764 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:53:44.764 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:53:44.772 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:53:44.775 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:53:45.725 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/715849107 2023-10-17T00:53:45.725 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:53:45.726 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504025 2023-10-17T00:53:45.726 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:53:45.726 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/715849107 ]] 2023-10-17T00:53:45.727 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/715849107 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\7\1\5\8\4\9\1\0\7 ]] 2023-10-17T00:53:45.727 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:53:55.727 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:53:55.730 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:53:55.735 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:53:55.747 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:53:56.586 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/715849107 2023-10-17T00:53:56.586 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:53:56.586 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504036 2023-10-17T00:53:56.587 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:53:56.587 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/715849107 ]] 2023-10-17T00:53:56.587 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/715849107 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\7\1\5\8\4\9\1\0\7 ]] 2023-10-17T00:53:56.587 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:54:00.001 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.003+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.002 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.003+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4bcd400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.002 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.003+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.003 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.003+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4acd300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.005 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.007+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.006 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.007+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4d33200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.016 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.015+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.016 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.015+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4384700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.033 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.035+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.033 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.035+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4d33480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.034 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.035+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.034 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.035+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4408600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.040 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.039+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.041 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.039+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4409900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.044 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.043+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.044 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.043+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4409780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.051 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.051+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.051 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.051+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4409e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.059 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.059+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.060 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.059+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.060 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.059+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4409b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.060 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.059+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4409d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.062 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.063+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.062 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.063+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4408100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.068 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.067+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.068 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.067+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4408380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.071 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.071+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.072 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.071+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4408280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.075 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.075+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.075 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.075+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4409880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.083 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.083+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.084 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.083+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4409680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.084 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.083+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.085 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.083+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4d33c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.090 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.091+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.090 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.091+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4409b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.092 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.091+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.093 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.091+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4409d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.095 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.095+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:00.095 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:54:00.095+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4408180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:06.592 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:54:06.592 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:54:06.593 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:54:06.609 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:54:07.540 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/715849107 2023-10-17T00:54:07.540 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:54:07.541 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504047 2023-10-17T00:54:07.542 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:54:07.542 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/715849107 ]] 2023-10-17T00:54:07.542 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/715849107 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\7\1\5\8\4\9\1\0\7 ]] 2023-10-17T00:54:07.542 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:54:17.557 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:54:17.561 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:54:17.561 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:54:17.577 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:54:18.431 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/715849107 2023-10-17T00:54:18.431 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:54:18.432 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504058 2023-10-17T00:54:18.432 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:54:18.432 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/715849107 ]] 2023-10-17T00:54:18.432 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/715849107 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\7\1\5\8\4\9\1\0\7 ]] 2023-10-17T00:54:18.433 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:54:28.441 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:54:28.445 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:54:28.461 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:54:28.462 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:54:29.389 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/715849107 2023-10-17T00:54:29.391 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:54:29.393 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504069 2023-10-17T00:54:29.394 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:54:29.394 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/715849107 ]] 2023-10-17T00:54:29.394 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/715849107 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\7\1\5\8\4\9\1\0\7 ]] 2023-10-17T00:54:29.394 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:54:39.396 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:54:39.397 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:54:39.400 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:54:39.404 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:54:40.238 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1212236727 2023-10-17T00:54:40.238 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:54:40.239 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504080 2023-10-17T00:54:40.239 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:54:40.239 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1212236727 ]] 2023-10-17T00:54:40.239 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1212236727 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\7\1\5\8\4\9\1\0\7 ]] 2023-10-17T00:54:40.240 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist add 172.21.15.169:0/1212236727 2023-10-17T00:54:41.953 INFO:tasks.workunit.client.0.smithi169.stderr:blocklisting 172.21.15.169:0/1212236727 until 2023-10-17T01:54:41.176072+0000 (3600 sec) 2023-10-17T00:54:42.002 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist ls 2023-10-17T00:54:42.008 INFO:tasks.workunit.client.0.smithi169.stderr:+ grep -q 172.21.15.169:0/1212236727 2023-10-17T00:54:42.844 INFO:tasks.workunit.client.0.smithi169.stderr:listed 10 entries 2023-10-17T00:54:42.894 INFO:tasks.workunit.client.0.smithi169.stderr:+ PREV_CLIENT_ADDR=172.21.15.169:0/1212236727 2023-10-17T00:54:42.894 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:54:52.903 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:54:52.903 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:54:52.903 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:54:52.907 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:54:53.759 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1212236727 2023-10-17T00:54:53.759 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:54:53.760 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504093 2023-10-17T00:54:53.760 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:54:53.760 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1212236727 ]] 2023-10-17T00:54:53.761 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1212236727 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\2\1\2\2\3\6\7\2\7 ]] 2023-10-17T00:54:53.761 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:54:59.998 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.002+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:54:59.998 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.002+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4d5ce00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.000 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.002+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.000 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.002+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4d5d000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.003 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.006+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.003 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.006+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4d5d200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.010 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.014+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.010 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.014+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4d5d400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.015 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.018+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.016 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.018+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4d5d600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.020 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.026+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.021 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.026+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4d5d800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.023 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.026+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.023 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.026+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4d5de00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.026 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.030+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.027 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.030+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4d5dc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.030 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.034+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.030 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.034+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4d5da00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.034 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.038+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.035 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.038+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4dfa000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.038 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.042+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.039 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.042+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4dfa200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.043 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.046+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.043 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.046+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4dfa400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.047 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.050+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.047 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.050+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4dfa600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.051 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.054+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.051 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.054+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4dfa800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.055 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.058+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.056 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.058+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4dfaa00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.059 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.062+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.060 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.062+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4dfac00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.064 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.066+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.064 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.066+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4dfae00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.069 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.070+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.069 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.070+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4dfb000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.073 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.078+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.073 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.078+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4dfb200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.077 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.082+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:00.077 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:55:00.082+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4dfb400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:55:03.780 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:55:03.783 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:55:03.783 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:55:03.784 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:55:04.727 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1212236727 2023-10-17T00:55:04.728 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:55:04.728 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504104 2023-10-17T00:55:04.729 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:55:04.729 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1212236727 ]] 2023-10-17T00:55:04.729 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1212236727 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\2\1\2\2\3\6\7\2\7 ]] 2023-10-17T00:55:04.729 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:55:14.730 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:55:14.731 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:55:14.731 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:55:14.742 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:55:15.686 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1212236727 2023-10-17T00:55:15.687 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:55:15.687 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504115 2023-10-17T00:55:15.687 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:55:15.687 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1212236727 ]] 2023-10-17T00:55:15.688 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1212236727 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\2\1\2\2\3\6\7\2\7 ]] 2023-10-17T00:55:15.688 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:55:25.688 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:55:25.689 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:55:25.700 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:55:25.716 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:55:26.592 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1212236727 2023-10-17T00:55:26.592 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:55:26.593 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504126 2023-10-17T00:55:26.593 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:55:26.593 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1212236727 ]] 2023-10-17T00:55:26.594 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1212236727 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\2\1\2\2\3\6\7\2\7 ]] 2023-10-17T00:55:26.594 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:55:36.595 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:55:36.596 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:55:36.597 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:55:36.623 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:55:37.510 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1212236727 2023-10-17T00:55:37.510 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:55:37.511 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504137 2023-10-17T00:55:37.511 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:55:37.512 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1212236727 ]] 2023-10-17T00:55:37.512 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1212236727 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\2\1\2\2\3\6\7\2\7 ]] 2023-10-17T00:55:37.512 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:55:47.513 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:55:47.514 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:55:47.533 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:55:47.540 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:55:48.427 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1212236727 2023-10-17T00:55:48.427 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:55:48.428 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504148 2023-10-17T00:55:48.428 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:55:48.429 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1212236727 ]] 2023-10-17T00:55:48.429 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1212236727 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\2\1\2\2\3\6\7\2\7 ]] 2023-10-17T00:55:48.429 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:55:57.548 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:55:57.549+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T00:55:58.432 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:55:58.441 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:55:58.448 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:55:58.458 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:55:59.423 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/144294311 2023-10-17T00:55:59.424 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:55:59.425 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504159 2023-10-17T00:55:59.425 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:55:59.425 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/144294311 ]] 2023-10-17T00:55:59.425 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/144294311 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\2\1\2\2\3\6\7\2\7 ]] 2023-10-17T00:55:59.426 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist add 172.21.15.169:0/144294311 2023-10-17T00:56:00.004 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.005+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.005 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.005+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4d32e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.005 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.009+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.005 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.009+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4acdf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.008 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.013+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.009 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.013+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4acc000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.011 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.013+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.011 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.013+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4accb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.013 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.017+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.013 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.017+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4e16480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.017 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.021+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.018 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.021+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4e16000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.021 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.025+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.021 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.025+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4e16d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.030 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.033+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.030 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.033+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4e16b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.034 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.037+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.034 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.037+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4e17280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.038 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.041+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.039 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.041+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4e17100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.041 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.045+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.042 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.045+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4e16f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.046 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.049+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.046 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.049+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4e16980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.050 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.053+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.050 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.053+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4e17500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.054 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.057+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.054 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.057+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4e17700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.059 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.061+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.059 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.061+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4e17900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.062 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.065+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.063 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.065+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4e17b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.067 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.069+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.067 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.069+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4e17d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.071 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.073+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.071 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.073+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4e17f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.076 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.077+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.076 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.077+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4e52100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.081 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.085+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.082 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:56:00.085+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4e52300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:56:00.934 INFO:tasks.workunit.client.0.smithi169.stderr:blocklisting 172.21.15.169:0/144294311 until 2023-10-17T01:56:00.182634+0000 (3600 sec) 2023-10-17T00:56:00.977 INFO:tasks.workunit.client.0.smithi169.stderr:+ grep -q 172.21.15.169:0/144294311 2023-10-17T00:56:00.978 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist ls 2023-10-17T00:56:01.665 INFO:tasks.workunit.client.0.smithi169.stderr:listed 11 entries 2023-10-17T00:56:01.692 INFO:tasks.workunit.client.0.smithi169.stderr:+ PREV_CLIENT_ADDR=172.21.15.169:0/144294311 2023-10-17T00:56:01.692 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:56:04.300 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:56:04.301+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T00:56:04.300 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:56:04.301+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T00:56:04.301 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:56:04.301+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T00:56:04.302 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:56:04.305+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T00:56:11.693 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:56:11.696 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:56:11.707 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:56:11.707 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:56:12.566 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/144294311 2023-10-17T00:56:12.567 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:56:12.568 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504172 2023-10-17T00:56:12.568 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:56:12.568 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/144294311 ]] 2023-10-17T00:56:12.568 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/144294311 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\4\4\2\9\4\3\1\1 ]] 2023-10-17T00:56:12.568 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:56:22.570 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:56:22.574 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:56:22.576 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:56:22.577 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:56:23.460 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/144294311 2023-10-17T00:56:23.461 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:56:23.461 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504183 2023-10-17T00:56:23.462 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:56:23.462 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/144294311 ]] 2023-10-17T00:56:23.462 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/144294311 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\4\4\2\9\4\3\1\1 ]] 2023-10-17T00:56:23.462 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:56:33.463 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:56:33.464 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:56:33.464 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:56:33.472 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:56:34.255 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/144294311 2023-10-17T00:56:34.255 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:56:34.256 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504194 2023-10-17T00:56:34.256 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:56:34.257 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/144294311 ]] 2023-10-17T00:56:34.257 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/144294311 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\4\4\2\9\4\3\1\1 ]] 2023-10-17T00:56:34.257 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:56:44.276 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:56:44.277 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:56:44.277 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:56:44.283 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:56:45.282 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/144294311 2023-10-17T00:56:45.282 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:56:45.283 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504205 2023-10-17T00:56:45.283 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:56:45.283 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/144294311 ]] 2023-10-17T00:56:45.284 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/144294311 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\4\4\2\9\4\3\1\1 ]] 2023-10-17T00:56:45.284 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:56:55.285 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:56:55.286 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:56:55.287 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:56:55.300 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:56:56.187 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/144294311 2023-10-17T00:56:56.188 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:56:56.188 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504216 2023-10-17T00:56:56.189 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:56:56.189 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/144294311 ]] 2023-10-17T00:56:56.189 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/144294311 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\4\4\2\9\4\3\1\1 ]] 2023-10-17T00:56:56.189 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:56:56.795 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:56:56.792+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T00:56:56.796 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:56:56.796+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T00:56:56.796 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:56:56.796+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T00:57:01.386 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:01.388+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T00:57:01.386 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:01.388+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T00:57:01.389 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:01.392+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T00:57:01.389 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:01.392+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T00:57:01.389 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:01.392+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T00:57:01.390 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:01.392+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T00:57:01.390 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:01.392+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T00:57:01.391 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:01.392+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T00:57:01.391 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:01.392+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T00:57:01.391 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:01.392+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T00:57:01.391 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:01.392+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T00:57:01.391 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:01.392+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T00:57:01.522 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:01.524+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c8410 2023-10-17T00:57:01.523 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:01.524+0000 7f31dd33a640 -1 reset not still connected to 0x55e9108bfe10 2023-10-17T00:57:01.523 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:01.524+0000 7f31dd33a640 -1 reset not still connected to 0x55e9111d0680 2023-10-17T00:57:01.523 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:01.524+0000 7f31dd33a640 -1 reset not still connected to 0x55e9189a5ba0 2023-10-17T00:57:01.523 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:01.524+0000 7f31dd33a640 -1 reset not still connected to 0x55e9189fd520 2023-10-17T00:57:01.523 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:01.524+0000 7f31dd33a640 -1 reset not still connected to 0x55e9251c5ad0 2023-10-17T00:57:01.524 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:01.524+0000 7f31dd33a640 -1 reset not still connected to 0x55e9251c5c70 2023-10-17T00:57:01.524 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:01.524+0000 7f31dd33a640 -1 reset not still connected to 0x55e9251c5d40 2023-10-17T00:57:01.524 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:01.524+0000 7f31dd33a640 -1 reset not still connected to 0x55e92cda3e10 2023-10-17T00:57:06.190 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:57:06.190 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:57:06.193 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:57:06.202 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:57:07.015 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1523367413 2023-10-17T00:57:07.016 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:57:07.016 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504227 2023-10-17T00:57:07.016 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:57:07.017 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1523367413 ]] 2023-10-17T00:57:07.017 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1523367413 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\4\4\2\9\4\3\1\1 ]] 2023-10-17T00:57:07.017 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist add 172.21.15.169:0/1523367413 2023-10-17T00:57:08.949 INFO:tasks.workunit.client.0.smithi169.stderr:blocklisting 172.21.15.169:0/1523367413 until 2023-10-17T01:57:08.135102+0000 (3600 sec) 2023-10-17T00:57:08.978 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist ls 2023-10-17T00:57:08.980 INFO:tasks.workunit.client.0.smithi169.stderr:+ grep -q 172.21.15.169:0/1523367413 2023-10-17T00:57:09.611 INFO:tasks.workunit.client.0.smithi169.stderr:listed 12 entries 2023-10-17T00:57:09.638 INFO:tasks.workunit.client.0.smithi169.stderr:+ PREV_CLIENT_ADDR=172.21.15.169:0/1523367413 2023-10-17T00:57:09.638 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:57:19.640 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:57:19.640 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:57:19.641 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:57:19.641 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:57:20.529 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1523367413 2023-10-17T00:57:20.529 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:57:20.530 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504240 2023-10-17T00:57:20.530 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:57:20.530 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1523367413 ]] 2023-10-17T00:57:20.530 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1523367413 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\5\2\3\3\6\7\4\1\3 ]] 2023-10-17T00:57:20.531 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:57:30.533 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:57:30.538 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:57:30.541 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:57:30.542 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:57:31.515 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1523367413 2023-10-17T00:57:31.516 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:57:31.518 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504251 2023-10-17T00:57:31.518 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:57:31.518 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1523367413 ]] 2023-10-17T00:57:31.518 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1523367413 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\5\2\3\3\6\7\4\1\3 ]] 2023-10-17T00:57:31.519 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:57:41.529 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:57:41.532 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:57:41.532 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:57:41.539 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:57:42.481 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1523367413 2023-10-17T00:57:42.481 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:57:42.482 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504262 2023-10-17T00:57:42.482 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:57:42.482 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1523367413 ]] 2023-10-17T00:57:42.483 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1523367413 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\5\2\3\3\6\7\4\1\3 ]] 2023-10-17T00:57:42.483 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:57:42.979 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:42.979+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T00:57:42.979 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:42.979+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T00:57:42.980 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:42.979+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbda750 2023-10-17T00:57:42.980 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:42.979+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T00:57:42.980 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:42.979+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T00:57:42.980 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:42.979+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T00:57:42.980 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:42.979+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T00:57:42.981 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:42.979+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T00:57:42.982 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:42.979+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T00:57:42.982 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:42.979+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T00:57:42.982 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:42.979+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T00:57:42.982 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:42.979+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T00:57:42.983 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:42.979+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T00:57:42.983 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:42.979+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T00:57:42.983 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:42.979+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T00:57:48.694 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:48.695+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T00:57:48.701 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:48.699+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T00:57:52.485 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:57:52.486 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:57:52.487 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:57:52.488 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:57:53.333 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1523367413 2023-10-17T00:57:53.333 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:57:53.334 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504273 2023-10-17T00:57:53.334 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:57:53.334 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1523367413 ]] 2023-10-17T00:57:53.335 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1523367413 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\5\2\3\3\6\7\4\1\3 ]] 2023-10-17T00:57:53.335 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:57:55.327 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:55.323+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T00:57:55.327 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:55.323+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T00:57:55.328 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:57:55.323+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T00:58:03.336 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:58:03.339 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:58:03.340 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:58:03.344 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:58:04.189 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1523367413 2023-10-17T00:58:04.190 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:58:04.190 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504284 2023-10-17T00:58:04.190 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:58:04.191 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1523367413 ]] 2023-10-17T00:58:04.191 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1523367413 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\5\2\3\3\6\7\4\1\3 ]] 2023-10-17T00:58:04.191 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:58:08.070 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:58:08.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T00:58:08.071 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:58:08.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T00:58:08.071 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:58:08.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbda750 2023-10-17T00:58:08.071 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:58:08.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T00:58:08.072 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:58:08.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T00:58:08.072 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:58:08.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T00:58:08.072 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:58:08.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T00:58:08.072 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:58:08.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T00:58:08.072 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:58:08.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T00:58:08.072 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:58:08.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T00:58:08.073 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:58:08.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T00:58:08.073 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:58:08.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T00:58:08.073 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:58:08.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T00:58:08.073 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:58:08.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T00:58:08.074 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:58:08.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T00:58:14.192 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:58:14.193 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:58:14.205 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:58:14.206 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:58:15.138 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/2169492734 2023-10-17T00:58:15.138 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:58:15.144 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504295 2023-10-17T00:58:15.144 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:58:15.144 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/2169492734 ]] 2023-10-17T00:58:15.145 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/2169492734 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\5\2\3\3\6\7\4\1\3 ]] 2023-10-17T00:58:15.145 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist add 172.21.15.169:0/2169492734 2023-10-17T00:58:16.971 INFO:tasks.workunit.client.0.smithi169.stderr:blocklisting 172.21.15.169:0/2169492734 until 2023-10-17T01:58:16.177757+0000 (3600 sec) 2023-10-17T00:58:17.009 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist ls 2023-10-17T00:58:17.009 INFO:tasks.workunit.client.0.smithi169.stderr:+ grep -q 172.21.15.169:0/2169492734 2023-10-17T00:58:17.740 INFO:tasks.workunit.client.0.smithi169.stderr:listed 13 entries 2023-10-17T00:58:17.765 INFO:tasks.workunit.client.0.smithi169.stderr:+ PREV_CLIENT_ADDR=172.21.15.169:0/2169492734 2023-10-17T00:58:17.765 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:58:27.766 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:58:27.772 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:58:27.780 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:58:27.792 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:58:28.677 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/2169492734 2023-10-17T00:58:28.677 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:58:28.681 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504308 2023-10-17T00:58:28.681 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:58:28.681 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/2169492734 ]] 2023-10-17T00:58:28.681 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/2169492734 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\2\1\6\9\4\9\2\7\3\4 ]] 2023-10-17T00:58:28.682 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:58:38.694 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:58:38.694 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:58:38.695 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:58:38.696 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:58:39.669 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/2169492734 2023-10-17T00:58:39.670 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:58:39.671 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504319 2023-10-17T00:58:39.671 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:58:39.671 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/2169492734 ]] 2023-10-17T00:58:39.671 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/2169492734 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\2\1\6\9\4\9\2\7\3\4 ]] 2023-10-17T00:58:39.671 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:58:49.674 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:58:49.674 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:58:49.675 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:58:49.677 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:58:50.499 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/2169492734 2023-10-17T00:58:50.500 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:58:50.507 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504330 2023-10-17T00:58:50.507 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:58:50.507 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/2169492734 ]] 2023-10-17T00:58:50.508 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/2169492734 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\2\1\6\9\4\9\2\7\3\4 ]] 2023-10-17T00:58:50.508 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:58:58.428 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:58:58.429+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T00:58:58.428 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:58:58.429+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T00:58:58.428 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:58:58.429+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbda750 2023-10-17T00:58:58.429 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:58:58.429+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T00:58:58.429 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:58:58.429+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T00:58:58.429 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:58:58.429+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T00:58:58.429 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:58:58.429+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T00:58:58.429 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:58:58.429+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T00:58:58.430 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:58:58.429+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T00:58:58.430 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:58:58.429+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T00:58:58.430 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:58:58.429+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T00:58:58.430 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:58:58.429+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T00:58:58.430 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:58:58.429+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T00:58:58.431 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:58:58.429+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T00:58:58.431 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:58:58.429+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T00:58:59.999 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.001+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:58:59.999 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.001+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4e53000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.001 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.005+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.001 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.005+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4e52e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.005 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.009+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.005 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.009+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4fee000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.016 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.017+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.016 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.017+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.017 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.017+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4fee200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.017 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.017+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4fee400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.018 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.021+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.018 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.021+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4fee600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.025 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.029+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.025 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.029+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4feec00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.026 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.029+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.026 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.029+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4feea80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.028 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.033+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.029 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.033+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4fef180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.037 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.041+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.037 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.041+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4fef000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.047 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.049+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.047 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.049+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4feee00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.051 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.053+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.051 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.053+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4fee880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.055 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.057+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.055 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.057+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4fef400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.058 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.061+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.058 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.061+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4fef600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.062 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.065+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.062 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.065+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4fef800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.067 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.069+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.067 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.069+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4fefa00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.072 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.073+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.073 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.073+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4fefc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.078 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.081+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.078 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.081+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4fefe00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.084 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.089+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.085 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.089+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b5030000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.089 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.093+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.090 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T00:59:00.093+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b5030200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T00:59:00.509 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:59:00.512 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:59:00.516 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:59:00.531 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:59:01.534 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/2169492734 2023-10-17T00:59:01.535 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:59:01.535 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504341 2023-10-17T00:59:01.536 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:59:01.536 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/2169492734 ]] 2023-10-17T00:59:01.536 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/2169492734 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\2\1\6\9\4\9\2\7\3\4 ]] 2023-10-17T00:59:01.536 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:59:11.551 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:59:11.551 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:59:11.552 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:59:11.559 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:59:12.456 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/2169492734 2023-10-17T00:59:12.456 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:59:12.551 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504352 2023-10-17T00:59:12.552 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:59:12.552 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/2169492734 ]] 2023-10-17T00:59:12.552 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/2169492734 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\2\1\6\9\4\9\2\7\3\4 ]] 2023-10-17T00:59:12.552 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:59:22.565 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:59:22.565 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:59:22.567 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:59:22.580 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:59:23.641 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1004139816 2023-10-17T00:59:23.641 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:59:23.645 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504363 2023-10-17T00:59:23.645 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:59:23.646 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1004139816 ]] 2023-10-17T00:59:23.646 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1004139816 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\2\1\6\9\4\9\2\7\3\4 ]] 2023-10-17T00:59:23.646 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist add 172.21.15.169:0/1004139816 2023-10-17T00:59:25.770 INFO:tasks.workunit.client.0.smithi169.stderr:blocklisting 172.21.15.169:0/1004139816 until 2023-10-17T01:59:24.970217+0000 (3600 sec) 2023-10-17T00:59:25.812 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist ls 2023-10-17T00:59:25.813 INFO:tasks.workunit.client.0.smithi169.stderr:+ grep -q 172.21.15.169:0/1004139816 2023-10-17T00:59:26.702 INFO:tasks.workunit.client.0.smithi169.stderr:listed 14 entries 2023-10-17T00:59:26.805 INFO:tasks.workunit.client.0.smithi169.stderr:+ PREV_CLIENT_ADDR=172.21.15.169:0/1004139816 2023-10-17T00:59:26.806 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:59:28.154 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:59:28.157+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T00:59:28.155 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:59:28.157+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T00:59:28.155 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:59:28.157+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T00:59:28.155 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:59:28.157+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T00:59:28.156 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:59:28.157+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T00:59:28.157 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:59:28.157+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T00:59:28.157 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:59:28.157+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T00:59:28.157 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:59:28.157+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T00:59:28.157 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:59:28.157+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T00:59:28.158 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:59:28.157+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T00:59:28.158 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:59:28.157+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T00:59:28.158 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:59:28.157+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T00:59:28.158 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:59:28.157+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T00:59:28.158 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T00:59:28.157+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T00:59:36.911 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:59:36.913 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:59:36.914 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:59:36.916 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:59:37.964 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1004139816 2023-10-17T00:59:37.964 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:59:37.965 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504377 2023-10-17T00:59:37.965 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:59:37.965 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1004139816 ]] 2023-10-17T00:59:37.966 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1004139816 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\0\0\4\1\3\9\8\1\6 ]] 2023-10-17T00:59:37.966 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:59:47.973 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:59:47.973 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:59:47.991 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:59:47.994 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:59:48.869 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1004139816 2023-10-17T00:59:48.869 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:59:48.870 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504388 2023-10-17T00:59:48.870 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:59:48.883 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1004139816 ]] 2023-10-17T00:59:48.883 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1004139816 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\0\0\4\1\3\9\8\1\6 ]] 2023-10-17T00:59:48.883 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T00:59:58.886 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T00:59:58.905 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T00:59:58.908 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T00:59:58.914 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T00:59:59.900 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1004139816 2023-10-17T00:59:59.900 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T00:59:59.901 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504399 2023-10-17T00:59:59.901 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T00:59:59.902 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1004139816 ]] 2023-10-17T00:59:59.902 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1004139816 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\0\0\4\1\3\9\8\1\6 ]] 2023-10-17T00:59:59.902 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:00:00.000 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.000+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.000 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.000+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4acc600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.001 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.004+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.001 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.004+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4d5c180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.006 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.008+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.006 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.008+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4bcda80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.009 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.012+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.010 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.012+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4d5c680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.012 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.016+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.012 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.016+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4bcdc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.014 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.016+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.015 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.016+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b3abd980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.018 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.020+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.018 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.020+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4408480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.020 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.024+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.021 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.024+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4408c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.030 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.032+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.030 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.032+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4e52680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.033 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.036+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.033 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.036+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b49c5400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.042 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.044+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.043 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.044+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b49c5580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.046 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.048+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.047 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.048+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4408200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.051 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.052+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.051 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.052+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4d33900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.055 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.056+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.055 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.056+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4d33f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.061 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.064+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.061 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.064+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4e53180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.064 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.068+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.065 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.068+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4e16700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.069 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.072+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.069 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.072+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4e16580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.073 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.076+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.073 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.076+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4fefd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.076 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.076+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.076 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.076+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4d33600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.080 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.080+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:00.080 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:00.080+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4ad5380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:00:09.902 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:00:09.904 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:00:09.905 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:00:09.908 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:00:10.922 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1004139816 2023-10-17T01:00:10.923 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:00:10.923 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504410 2023-10-17T01:00:10.923 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:00:10.924 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1004139816 ]] 2023-10-17T01:00:10.924 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1004139816 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\0\0\4\1\3\9\8\1\6 ]] 2023-10-17T01:00:10.924 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:00:20.925 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:00:20.926 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:00:20.932 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:00:20.933 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:00:21.965 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1004139816 2023-10-17T01:00:21.966 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:00:21.967 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504421 2023-10-17T01:00:21.967 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:00:21.967 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1004139816 ]] 2023-10-17T01:00:21.968 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1004139816 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\0\0\4\1\3\9\8\1\6 ]] 2023-10-17T01:00:21.968 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:00:31.974 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:00:31.975 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:00:31.981 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:00:31.990 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:00:32.963 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/2965545755 2023-10-17T01:00:32.964 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:00:32.964 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504432 2023-10-17T01:00:32.964 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:00:32.965 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/2965545755 ]] 2023-10-17T01:00:32.965 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/2965545755 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\0\0\4\1\3\9\8\1\6 ]] 2023-10-17T01:00:32.965 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist add 172.21.15.169:0/2965545755 2023-10-17T01:00:34.790 INFO:tasks.workunit.client.0.smithi169.stderr:blocklisting 172.21.15.169:0/2965545755 until 2023-10-17T02:00:34.034419+0000 (3600 sec) 2023-10-17T01:00:34.840 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist ls 2023-10-17T01:00:34.841 INFO:tasks.workunit.client.0.smithi169.stderr:+ grep -q 172.21.15.169:0/2965545755 2023-10-17T01:00:35.590 INFO:tasks.workunit.client.0.smithi169.stderr:listed 15 entries 2023-10-17T01:00:35.620 INFO:tasks.workunit.client.0.smithi169.stderr:+ PREV_CLIENT_ADDR=172.21.15.169:0/2965545755 2023-10-17T01:00:35.620 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:00:45.622 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:00:45.623 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:00:45.627 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:00:45.632 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:00:46.662 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/2965545755 2023-10-17T01:00:46.662 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:00:46.663 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504446 2023-10-17T01:00:46.663 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:00:46.664 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/2965545755 ]] 2023-10-17T01:00:46.664 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/2965545755 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\2\9\6\5\5\4\5\7\5\5 ]] 2023-10-17T01:00:46.664 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:00:56.672 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:00:56.672 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:00:56.673 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:00:56.678 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:00:57.608 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/2965545755 2023-10-17T01:00:57.609 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:00:57.610 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504457 2023-10-17T01:00:57.610 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:00:57.610 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/2965545755 ]] 2023-10-17T01:00:57.610 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/2965545755 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\2\9\6\5\5\4\5\7\5\5 ]] 2023-10-17T01:00:57.611 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:01:00.001 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:59.999+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.001 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:59.999+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4dfbb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.001 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:59.999+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.002 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:00:59.999+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b46a5280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.005 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.007+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.006 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.007+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b3e34e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.009 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.011+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.009 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.011+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4a3ff80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.015 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.015+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.016 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.015+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4e52280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.020 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.019+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.020 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.019+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4e52680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.023 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.023+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.023 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.023+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4d32b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.026 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.027+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.026 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.027+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4d33580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.030 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.031+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.030 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.031+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4e17f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.036 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.035+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.036 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.035+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4e16d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.045 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.047+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.045 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.047+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4ad5100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.046 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.047+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.046 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.047+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4d33a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.050 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.051+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.050 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.051+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b49c5a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.051 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.051+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.051 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.051+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4e16c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.059 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.059+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.059 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.059+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4e17c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.062 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.063+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.063 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.063+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b5031800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.069 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.071+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.070 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.071+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b5031600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.075 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.075+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.076 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.075+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b5031400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.079 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.079+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.079 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.079+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b5030a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.082 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.083+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:00.082 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:01:00.083+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b5030c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:01:07.616 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:01:07.617 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:01:07.624 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:01:07.629 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:01:08.671 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/2965545755 2023-10-17T01:01:08.671 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:01:08.672 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504468 2023-10-17T01:01:08.672 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:01:08.673 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/2965545755 ]] 2023-10-17T01:01:08.673 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/2965545755 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\2\9\6\5\5\4\5\7\5\5 ]] 2023-10-17T01:01:08.673 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:01:18.674 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:01:18.675 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:01:18.681 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:01:18.686 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:01:19.499 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/2965545755 2023-10-17T01:01:19.499 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:01:19.500 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504479 2023-10-17T01:01:19.500 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:01:19.501 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/2965545755 ]] 2023-10-17T01:01:19.501 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/2965545755 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\2\9\6\5\5\4\5\7\5\5 ]] 2023-10-17T01:01:19.501 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:01:20.961 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:01:20.950+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:01:20.962 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:01:20.954+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:01:20.962 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:01:20.954+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:01:29.502 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:01:29.503 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:01:29.513 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:01:29.521 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:01:30.479 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/2965545755 2023-10-17T01:01:30.484 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:01:30.485 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504490 2023-10-17T01:01:30.486 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:01:30.486 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/2965545755 ]] 2023-10-17T01:01:30.486 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/2965545755 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\2\9\6\5\5\4\5\7\5\5 ]] 2023-10-17T01:01:30.486 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:01:33.165 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:01:33.158+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:01:33.166 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:01:33.158+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:01:33.166 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:01:33.162+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:01:40.487 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:01:40.488 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:01:40.490 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:01:40.500 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:01:41.593 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/2965545755 2023-10-17T01:01:41.593 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:01:41.594 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504501 2023-10-17T01:01:41.594 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:01:41.594 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/2965545755 ]] 2023-10-17T01:01:41.595 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/2965545755 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\2\9\6\5\5\4\5\7\5\5 ]] 2023-10-17T01:01:41.595 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:01:51.599 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:01:51.599 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:01:51.599 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:01:51.611 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:01:52.862 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/471946586 2023-10-17T01:01:52.862 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:01:52.863 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504512 2023-10-17T01:01:52.863 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:01:52.864 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/471946586 ]] 2023-10-17T01:01:52.864 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/471946586 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\2\9\6\5\5\4\5\7\5\5 ]] 2023-10-17T01:01:52.864 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist add 172.21.15.169:0/471946586 2023-10-17T01:01:54.790 INFO:tasks.workunit.client.0.smithi169.stderr:blocklisting 172.21.15.169:0/471946586 until 2023-10-17T02:01:54.054634+0000 (3600 sec) 2023-10-17T01:01:54.838 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist ls 2023-10-17T01:01:54.849 INFO:tasks.workunit.client.0.smithi169.stderr:+ grep -q 172.21.15.169:0/471946586 2023-10-17T01:01:55.702 INFO:tasks.workunit.client.0.smithi169.stderr:listed 16 entries 2023-10-17T01:01:55.728 INFO:tasks.workunit.client.0.smithi169.stderr:+ PREV_CLIENT_ADDR=172.21.15.169:0/471946586 2023-10-17T01:01:55.728 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:02:00.001 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.006+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.001 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.006+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4acd500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.001 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.006+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.002 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.006+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4d32e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.005 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.010+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.005 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.010+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4a3fd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.012 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.018+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.012 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.018+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4fef200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.014 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.018+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.014 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.018+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4dfbc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.018 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.022+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.019 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.022+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4409e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.041 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.046+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.041 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.046+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b5030480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.048 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.050+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.048 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.050+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b5031680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.064 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.066+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.065 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.066+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b5030e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.070 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.074+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.070 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.074+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b5031000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.074 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.078+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.074 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.078+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b5031200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.077 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.082+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.077 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.082+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4bcde00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.079 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.082+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.079 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.082+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b5030880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.084 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.086+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.084 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.086+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b5030380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.087 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.090+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.087 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.090+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b5030780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.091 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.094+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.091 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.094+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b5031a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.096 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.098+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.096 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.098+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b5031c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.099 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.102+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.099 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.102+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b5031e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.102 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.106+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.102 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.106+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b5046000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.105 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.110+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:00.105 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:02:00.110+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b5046200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:05.738 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:02:05.740 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:02:05.748 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:02:05.764 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:02:06.592 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/471946586 2023-10-17T01:02:06.593 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:02:06.593 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504526 2023-10-17T01:02:06.594 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:02:06.594 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/471946586 ]] 2023-10-17T01:02:06.594 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/471946586 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\4\7\1\9\4\6\5\8\6 ]] 2023-10-17T01:02:06.595 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:02:10.682 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:02:10.686+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:02:10.689 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:02:10.686+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:02:10.690 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:02:10.686+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:02:16.596 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:02:16.598 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:02:16.610 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:02:16.621 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:02:17.614 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/471946586 2023-10-17T01:02:17.615 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:02:17.615 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504537 2023-10-17T01:02:17.615 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:02:17.616 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/471946586 ]] 2023-10-17T01:02:17.616 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/471946586 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\4\7\1\9\4\6\5\8\6 ]] 2023-10-17T01:02:17.616 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:02:27.617 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:02:27.620 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:02:27.624 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:02:27.638 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:02:28.513 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/471946586 2023-10-17T01:02:28.514 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:02:28.515 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504548 2023-10-17T01:02:28.516 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:02:28.516 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/471946586 ]] 2023-10-17T01:02:28.516 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/471946586 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\4\7\1\9\4\6\5\8\6 ]] 2023-10-17T01:02:28.516 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:02:38.524 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:02:38.525 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:02:38.528 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:02:38.529 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:02:39.381 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/471946586 2023-10-17T01:02:39.381 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:02:39.382 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504559 2023-10-17T01:02:39.382 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:02:39.382 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/471946586 ]] 2023-10-17T01:02:39.383 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/471946586 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\4\7\1\9\4\6\5\8\6 ]] 2023-10-17T01:02:39.383 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:02:49.386 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:02:49.395 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:02:49.401 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:02:49.408 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:02:50.328 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/315948540 2023-10-17T01:02:50.329 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:02:50.329 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504570 2023-10-17T01:02:50.329 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:02:50.330 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/315948540 ]] 2023-10-17T01:02:50.330 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/315948540 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\4\7\1\9\4\6\5\8\6 ]] 2023-10-17T01:02:50.330 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist add 172.21.15.169:0/315948540 2023-10-17T01:02:52.087 INFO:tasks.workunit.client.0.smithi169.stderr:blocklisting 172.21.15.169:0/315948540 until 2023-10-17T02:02:51.322819+0000 (3600 sec) 2023-10-17T01:02:52.138 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist ls 2023-10-17T01:02:52.138 INFO:tasks.workunit.client.0.smithi169.stderr:+ grep -q 172.21.15.169:0/315948540 2023-10-17T01:02:52.876 INFO:tasks.workunit.client.0.smithi169.stderr:listed 17 entries 2023-10-17T01:02:52.924 INFO:tasks.workunit.client.0.smithi169.stderr:+ PREV_CLIENT_ADDR=172.21.15.169:0/315948540 2023-10-17T01:02:52.924 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:02:59.999 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.001+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:02:59.999 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.001+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4e16e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.001 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.005+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.002 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.005+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4384700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.005 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.009+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.005 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.009+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4acdd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.010 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.013+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.011 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.013+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4d5cc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.014 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.017+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.014 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.017+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4408b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.019 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.021+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.020 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.021+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4e52880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.023 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.025+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.023 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.025+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b5047700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.028 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.029+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.028 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.029+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b5047580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.030 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.033+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.030 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.033+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b5046f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.033 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.037+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.033 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.037+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b5047080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.036 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.041+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.037 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.041+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b5047280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.039 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.041+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.040 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.041+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b49c5a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.043 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.045+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.044 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.045+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b5046c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.048 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.049+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.048 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.049+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b5046880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.057 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.061+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.057 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.061+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b5046800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.061 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.065+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.061 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.065+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b5046600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.066 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.069+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.066 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.069+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b5046400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.070 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.073+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.071 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.073+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b5047900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.074 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.077+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.074 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.077+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b5047b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.078 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.081+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:00.079 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:03:00.081+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b5047d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:03:02.926 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:03:02.926 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:03:02.927 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:03:02.928 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:03:03.863 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/315948540 2023-10-17T01:03:03.863 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:03:03.864 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504583 2023-10-17T01:03:03.864 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:03:03.864 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/315948540 ]] 2023-10-17T01:03:03.865 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/315948540 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\3\1\5\9\4\8\5\4\0 ]] 2023-10-17T01:03:03.865 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:03:13.870 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:03:13.871 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:03:13.871 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:03:13.874 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:03:14.688 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/315948540 2023-10-17T01:03:14.698 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:03:14.699 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504594 2023-10-17T01:03:14.700 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:03:14.700 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/315948540 ]] 2023-10-17T01:03:14.700 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/315948540 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\3\1\5\9\4\8\5\4\0 ]] 2023-10-17T01:03:14.700 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:03:24.701 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:03:24.703 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:03:24.704 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:03:24.727 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:03:25.762 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/315948540 2023-10-17T01:03:25.763 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:03:25.763 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504605 2023-10-17T01:03:25.764 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:03:25.764 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/315948540 ]] 2023-10-17T01:03:25.764 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/315948540 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\3\1\5\9\4\8\5\4\0 ]] 2023-10-17T01:03:25.764 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:03:33.087 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:03:33.084+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:03:33.088 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:03:33.084+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:03:33.088 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:03:33.084+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:03:33.088 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:03:33.084+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:03:33.088 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:03:33.084+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:03:33.088 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:03:33.088+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:03:33.089 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:03:33.088+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:03:33.089 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:03:33.088+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:03:35.766 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:03:35.775 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:03:35.784 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:03:35.796 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:03:36.713 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/315948540 2023-10-17T01:03:36.714 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:03:36.714 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504616 2023-10-17T01:03:36.714 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:03:36.715 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/315948540 ]] 2023-10-17T01:03:36.715 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/315948540 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\3\1\5\9\4\8\5\4\0 ]] 2023-10-17T01:03:36.715 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:03:46.721 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:03:46.727 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:03:46.728 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:03:46.728 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:03:47.699 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/315948540 2023-10-17T01:03:47.700 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:03:47.701 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504627 2023-10-17T01:03:47.701 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:03:47.701 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/315948540 ]] 2023-10-17T01:03:47.701 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/315948540 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\3\1\5\9\4\8\5\4\0 ]] 2023-10-17T01:03:47.702 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:03:57.706 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:03:57.707 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:03:57.717 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:03:57.731 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:03:58.973 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/2992472699 2023-10-17T01:03:58.973 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:03:58.974 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504638 2023-10-17T01:03:58.974 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:03:58.974 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/2992472699 ]] 2023-10-17T01:03:58.975 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/2992472699 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\3\1\5\9\4\8\5\4\0 ]] 2023-10-17T01:03:58.975 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist add 172.21.15.169:0/2992472699 2023-10-17T01:04:00.139 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.140+0000 7f34fd8c6640 -1 librbd::mirror::GetInfoRequest: 0x55c1b4782d80 handle_get_mirror_image: failed to retrieve mirroring state: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.140 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.140+0000 7f34fd8c6640 -1 librbd::io::AioCompletion: 0x55c1b521b8c0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.141 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.144+0000 7f34fd8c6640 -1 librbd::ImageWatcher: 0x55c1b4e62900 image watch failed: 94290452195328, (107) Transport endpoint is not connected 2023-10-17T01:04:00.142 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.144+0000 7f34fd8c6640 -1 librbd::Watcher: 0x55c1b4e62900 handle_error: handle=94290452195328: (107) Transport endpoint is not connected 2023-10-17T01:04:00.142 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.144+0000 7f34fd8c6640 -1 librbd::ImageWatcher: 0x55c1b4e62c00 image watch failed: 94290452200448, (107) Transport endpoint is not connected 2023-10-17T01:04:00.142 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.144+0000 7f34fd8c6640 -1 librbd::Watcher: 0x55c1b4e62c00 handle_error: handle=94290452200448: (107) Transport endpoint is not connected 2023-10-17T01:04:00.142 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.144+0000 7f34fd8c6640 -1 librbd::ImageWatcher: 0x55c1b51d6900 image watch failed: 94290456161280, (107) Transport endpoint is not connected 2023-10-17T01:04:00.142 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.144+0000 7f34fd8c6640 -1 librbd::Watcher: 0x55c1b51d6900 handle_error: handle=94290456161280: (107) Transport endpoint is not connected 2023-10-17T01:04:00.148 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.144+0000 7f34fd8c6640 -1 librbd::watcher::RewatchRequest: 0x55c1b47475e0 handle_unwatch client blocklisted 2023-10-17T01:04:00.148 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.144+0000 7f34fd8c6640 -1 librbd::Watcher: 0x55c1b4e62c00 handle_rewatch: client blocklisted 2023-10-17T01:04:00.149 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.144+0000 7f34fd8c6640 -1 librbd::watcher::RewatchRequest: 0x55c1b47c99f0 handle_unwatch client blocklisted 2023-10-17T01:04:00.149 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.144+0000 7f34fd8c6640 -1 librbd::Watcher: 0x55c1b4e62900 handle_rewatch: client blocklisted 2023-10-17T01:04:00.149 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.144+0000 7f34fd8c6640 -1 librbd::ManagedLock: 0x55c1b3f1e1b8 send_acquire_lock: watcher not registered - client blocklisted 2023-10-17T01:04:00.149 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.144+0000 7f34fd8c6640 -1 librbd::ManagedLock: 0x55c1b39f1058 send_acquire_lock: watcher not registered - client blocklisted 2023-10-17T01:04:00.149 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.144+0000 7f34fd8c6640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x55c1b4bed450 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.149 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.144+0000 7f34fd8c6640 -1 librbd::io::AioCompletion: 0x55c1b4ef49a0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.150 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.144+0000 7f34fd8c6640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x55c1b4bed380 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.150 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.144+0000 7f34fd8c6640 -1 librbd::io::AioCompletion: 0x55c1b4e15ce0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.150 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.152+0000 7f34fd0c5640 -1 librbd::ImageWatcher: 0x55c1b51d6c00 image watch failed: 94290456159232, (107) Transport endpoint is not connected 2023-10-17T01:04:00.150 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.152+0000 7f34fd0c5640 -1 librbd::Watcher: 0x55c1b51d6c00 handle_error: handle=94290456159232: (107) Transport endpoint is not connected 2023-10-17T01:04:00.151 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.152+0000 7f34fd0c5640 -1 librbd::watcher::RewatchRequest: 0x55c1b484f7c0 handle_unwatch client blocklisted 2023-10-17T01:04:00.151 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.152+0000 7f34fd0c5640 -1 librbd::Watcher: 0x55c1b51d6900 handle_rewatch: client blocklisted 2023-10-17T01:04:00.151 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.152+0000 7f34fd0c5640 -1 librbd::managed_lock::BreakRequest: 0x55c1b504d680 handle_get_watchers: failed to retrieve watchers: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.151 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.152+0000 7f34fd0c5640 -1 librbd::managed_lock::AcquireRequest: 0x55c1b51c2b40 handle_break_lock: failed to break lock : (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.151 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.152+0000 7f34fd0c5640 -1 librbd::ImageWatcher: 0x55c1b4e63200 image watch failed: 94290452194304, (107) Transport endpoint is not connected 2023-10-17T01:04:00.152 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.152+0000 7f34fd0c5640 -1 librbd::Watcher: 0x55c1b4e63200 handle_error: handle=94290452194304: (107) Transport endpoint is not connected 2023-10-17T01:04:00.152 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.152+0000 7f34fd0c5640 -1 librbd::ImageWatcher: 0x55c1b4e62f00 image watch failed: 94290456158208, (107) Transport endpoint is not connected 2023-10-17T01:04:00.152 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.152+0000 7f34fd0c5640 -1 librbd::Watcher: 0x55c1b4e62f00 handle_error: handle=94290456158208: (107) Transport endpoint is not connected 2023-10-17T01:04:00.152 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.152+0000 7f34fd0c5640 -1 librbd::ManagedLock: 0x55c1b4481398 send_acquire_lock: watcher not registered - client blocklisted 2023-10-17T01:04:00.152 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.152+0000 7f34fd0c5640 -1 librbd::ManagedLock: 0x55c1b4737398 handle_acquire_lock: failed to acquire exclusive lock: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.152 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.152+0000 7f34fd0c5640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x55c1b4bed5f0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.153 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.152+0000 7f34fd0c5640 -1 librbd::io::AioCompletion: 0x55c1b521ab00 fail: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.153 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.152+0000 7f34fd0c5640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x55c1b4bed860 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.153 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.152+0000 7f34fd0c5640 -1 librbd::io::AioCompletion: 0x55c1b5310000 fail: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.154 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.152+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.154 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.152+0000 7f34fd0c5640 -1 librbd::managed_lock::BreakRequest: 0x55c1b504de00 handle_get_watchers: failed to retrieve watchers: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.154 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.152+0000 7f34fd0c5640 -1 librbd::managed_lock::AcquireRequest: 0x55c1b51c2a50 handle_break_lock: failed to break lock : (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.154 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.152+0000 7f34fd0c5640 -1 librbd::ManagedLock: 0x55c1b3d591f8 handle_acquire_lock: failed to acquire exclusive lock: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.155 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.152+0000 7f34fd0c5640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x55c1b4bed6c0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.155 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.152+0000 7f34fd0c5640 -1 librbd::io::AioCompletion: 0x55c1b521af20 fail: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.155 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.152+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4fef680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.155 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.152+0000 7f34fd0c5640 -1 librbd::watcher::RewatchRequest: 0x55c1b484f7c0 handle_unwatch client blocklisted 2023-10-17T01:04:00.155 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.152+0000 7f34fd0c5640 -1 librbd::watcher::RewatchRequest: 0x55c1b484f770 handle_unwatch client blocklisted 2023-10-17T01:04:00.155 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.156+0000 7f34fd0c5640 -1 librbd::ImageWatcher: 0x55c1b51d6300 image watch failed: 94290456165376, (107) Transport endpoint is not connected 2023-10-17T01:04:00.156 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.156+0000 7f34fd0c5640 -1 librbd::Watcher: 0x55c1b51d6300 handle_error: handle=94290456165376: (107) Transport endpoint is not connected 2023-10-17T01:04:00.156 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.156+0000 7f34fd8c6640 -1 librbd::managed_lock::BreakRequest: 0x55c1b50728c0 handle_get_watchers: failed to retrieve watchers: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.156 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.156+0000 7f34fd8c6640 -1 librbd::managed_lock::AcquireRequest: 0x55c1b51c23c0 handle_break_lock: failed to break lock : (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.156 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.156+0000 7f34fd8c6640 -1 librbd::ImageWatcher: 0x55c1b51d6000 image watch failed: 94290452199424, (107) Transport endpoint is not connected 2023-10-17T01:04:00.156 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.156+0000 7f34fd8c6640 -1 librbd::Watcher: 0x55c1b51d6000 handle_error: handle=94290452199424: (107) Transport endpoint is not connected 2023-10-17T01:04:00.157 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.156+0000 7f34fd8c6640 -1 librbd::ManagedLock: 0x55c1b3b7a838 handle_acquire_lock: failed to acquire exclusive lock: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.157 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.156+0000 7f34fd8c6640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x55c1b4bed520 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.157 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.156+0000 7f34fd8c6640 -1 librbd::io::AioCompletion: 0x55c1b521b1e0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.157 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.156+0000 7f34fd8c6640 -1 librbd::managed_lock::BreakRequest: 0x55c1b504d540 handle_get_watchers: failed to retrieve watchers: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.157 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.156+0000 7f34fd8c6640 -1 librbd::managed_lock::AcquireRequest: 0x55c1b51c2ff0 handle_break_lock: failed to break lock : (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.157 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.156+0000 7f34fd8c6640 -1 librbd::ManagedLock: 0x55c1b44809d8 handle_acquire_lock: failed to acquire exclusive lock: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.158 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.156+0000 7f34fd8c6640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x55c1b4bed790 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.158 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.156+0000 7f34fd8c6640 -1 librbd::io::AioCompletion: 0x55c1b521b340 fail: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.158 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.156+0000 7f34fd8c6640 -1 librbd::watcher::RewatchRequest: 0x55c1b44ef8b0 handle_unwatch client blocklisted 2023-10-17T01:04:00.158 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.156+0000 7f34fd8c6640 -1 librbd::Watcher: 0x55c1b51d6300 handle_rewatch: client blocklisted 2023-10-17T01:04:00.158 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.156+0000 7f34fd8c6640 -1 librbd::watcher::RewatchRequest: 0x55c1b44eeaa0 handle_unwatch client blocklisted 2023-10-17T01:04:00.159 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.156+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.159 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.156+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4feff80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.159 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.160+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.160 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.160+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4fef680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.171 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.172+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.171 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.172+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b5030600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.179 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.180+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.179 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.180+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b5031c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.196 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.200+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.196 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.200+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4feff80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.209 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.212+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.209 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.212+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4fef680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.212 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.216+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.212 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.216+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b5030880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.214 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.216+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.214 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.216+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b5030600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.216 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.220+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.216 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.220+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b5031c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.580 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.584+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve initial metadata: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.581 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.584+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b3e34f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.686 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.688+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve initial metadata: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:00.686 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:04:00.688+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4ad5100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:04:01.126 INFO:tasks.workunit.client.0.smithi169.stderr:blocklisting 172.21.15.169:0/2992472699 until 2023-10-17T02:04:00.354812+0000 (3600 sec) 2023-10-17T01:04:01.164 INFO:tasks.workunit.client.0.smithi169.stderr:+ grep -q 172.21.15.169:0/2992472699 2023-10-17T01:04:01.170 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist ls 2023-10-17T01:04:01.886 INFO:tasks.workunit.client.0.smithi169.stderr:listed 18 entries 2023-10-17T01:04:01.930 INFO:tasks.workunit.client.0.smithi169.stderr:+ PREV_CLIENT_ADDR=172.21.15.169:0/2992472699 2023-10-17T01:04:01.930 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:04:11.958 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:04:11.959 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:04:11.963 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:04:11.997 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:04:12.412 INFO:tasks.workunit.client.0.smithi169.stderr:2023-10-17T01:04:12.416+0000 7fa97ffff640 0 --2- 172.21.15.169:0/2344090266 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa980070950 0x7fa9801bd4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-10-17T01:04:13.228 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/2992472699 2023-10-17T01:04:13.228 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:04:13.228 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504653 2023-10-17T01:04:13.229 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:04:13.229 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/2992472699 ]] 2023-10-17T01:04:13.229 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/2992472699 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\2\9\9\2\4\7\2\6\9\9 ]] 2023-10-17T01:04:13.229 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:04:23.233 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:04:23.234 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:04:23.234 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:04:23.239 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:04:24.175 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/2992472699 2023-10-17T01:04:24.175 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:04:24.176 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504664 2023-10-17T01:04:24.176 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:04:24.176 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/2992472699 ]] 2023-10-17T01:04:24.177 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/2992472699 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\2\9\9\2\4\7\2\6\9\9 ]] 2023-10-17T01:04:24.177 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:04:34.178 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:04:34.180 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:04:34.183 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:04:34.187 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:04:35.184 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/2992472699 2023-10-17T01:04:35.184 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:04:35.185 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504675 2023-10-17T01:04:35.185 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:04:35.185 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/2992472699 ]] 2023-10-17T01:04:35.185 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/2992472699 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\2\9\9\2\4\7\2\6\9\9 ]] 2023-10-17T01:04:35.186 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:04:45.189 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:04:45.189 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:04:45.190 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:04:45.191 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:04:46.082 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/2992472699 2023-10-17T01:04:46.082 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:04:46.082 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504686 2023-10-17T01:04:46.083 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:04:46.083 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/2992472699 ]] 2023-10-17T01:04:46.083 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/2992472699 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\2\9\9\2\4\7\2\6\9\9 ]] 2023-10-17T01:04:46.083 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:04:56.091 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:04:56.091 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:04:56.102 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:04:56.105 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:04:57.054 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/2992472699 2023-10-17T01:04:57.055 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:04:57.055 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504697 2023-10-17T01:04:57.056 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:04:57.056 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/2992472699 ]] 2023-10-17T01:04:57.056 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/2992472699 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\2\9\9\2\4\7\2\6\9\9 ]] 2023-10-17T01:04:57.056 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:05:07.057 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:05:07.058 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:05:07.060 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:05:07.064 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:05:07.986 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/2992472699 2023-10-17T01:05:07.986 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:05:07.986 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504707 2023-10-17T01:05:07.987 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:05:07.987 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/2992472699 ]] 2023-10-17T01:05:07.987 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/2992472699 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\2\9\9\2\4\7\2\6\9\9 ]] 2023-10-17T01:05:07.987 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:05:17.988 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:05:17.991 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:05:18.009 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:05:18.026 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:05:18.932 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1405644418 2023-10-17T01:05:18.933 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:05:18.939 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504718 2023-10-17T01:05:18.939 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:05:18.939 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1405644418 ]] 2023-10-17T01:05:18.940 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1405644418 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\2\9\9\2\4\7\2\6\9\9 ]] 2023-10-17T01:05:18.940 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist add 172.21.15.169:0/1405644418 2023-10-17T01:05:20.728 INFO:tasks.workunit.client.0.smithi169.stderr:blocklisting 172.21.15.169:0/1405644418 until 2023-10-17T02:05:19.934424+0000 (3600 sec) 2023-10-17T01:05:20.790 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist ls 2023-10-17T01:05:20.791 INFO:tasks.workunit.client.0.smithi169.stderr:+ grep -q 172.21.15.169:0/1405644418 2023-10-17T01:05:21.595 INFO:tasks.workunit.client.0.smithi169.stderr:listed 19 entries 2023-10-17T01:05:21.647 INFO:tasks.workunit.client.0.smithi169.stderr:+ PREV_CLIENT_ADDR=172.21.15.169:0/1405644418 2023-10-17T01:05:21.647 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:05:31.650 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:05:31.652 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:05:31.659 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:05:31.666 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:05:32.530 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1405644418 2023-10-17T01:05:32.530 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:05:32.540 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504732 2023-10-17T01:05:32.541 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:05:32.541 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1405644418 ]] 2023-10-17T01:05:32.541 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1405644418 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\4\0\5\6\4\4\4\1\8 ]] 2023-10-17T01:05:32.542 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:05:42.543 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:05:42.544 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:05:42.544 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:05:42.546 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:05:43.504 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1405644418 2023-10-17T01:05:43.504 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:05:43.505 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504743 2023-10-17T01:05:43.505 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:05:43.506 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1405644418 ]] 2023-10-17T01:05:43.506 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1405644418 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\4\0\5\6\4\4\4\1\8 ]] 2023-10-17T01:05:43.506 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:05:53.522 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:05:53.526 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:05:53.530 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:05:53.545 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:05:54.510 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1405644418 2023-10-17T01:05:54.510 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:05:54.511 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504754 2023-10-17T01:05:54.511 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:05:54.511 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1405644418 ]] 2023-10-17T01:05:54.511 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1405644418 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\4\0\5\6\4\4\4\1\8 ]] 2023-10-17T01:05:54.512 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:06:00.000 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.001+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.001 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.001+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4a3f900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.001 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.005+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.001 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.005+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4ad5100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.005 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.009+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.005 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.009+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4dfaf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.010 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.013+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.010 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.013+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4e16f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.021 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.025+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.021 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.025+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4dfad00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.030 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.033+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.030 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.033+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b5031680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.042 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.045+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.043 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.045+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b5031180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.046 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.049+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.047 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.049+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b5031100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.048 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.049+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.048 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.053+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b5046a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.059 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.061+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.060 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.061+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b5046100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.063 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.065+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.063 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.065+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b5046300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.066 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.069+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.067 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.069+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b5031b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.069 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.073+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.069 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.073+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4e52f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.076 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.081+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.077 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.081+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b5046580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.077 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.081+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.077 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.081+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b5047200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.079 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.081+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.079 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.081+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4d44a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.081 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.085+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.081 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.085+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4bcd880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.085 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.089+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.086 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.089+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4e52c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.090 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.093+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.090 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.093+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4d45980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.093 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.097+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:00.094 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:06:00.097+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4d45800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:06:04.513 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:06:04.514 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:06:04.514 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:06:04.540 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:06:05.500 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1405644418 2023-10-17T01:06:05.500 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:06:05.501 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504765 2023-10-17T01:06:05.501 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:06:05.501 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1405644418 ]] 2023-10-17T01:06:05.502 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1405644418 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\4\0\5\6\4\4\4\1\8 ]] 2023-10-17T01:06:05.502 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:06:15.506 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:06:15.507 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:06:15.508 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:06:15.514 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:06:16.411 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1405644418 2023-10-17T01:06:16.412 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:06:16.412 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504776 2023-10-17T01:06:16.412 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:06:16.413 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1405644418 ]] 2023-10-17T01:06:16.413 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1405644418 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\4\0\5\6\4\4\4\1\8 ]] 2023-10-17T01:06:16.413 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:06:26.417 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:06:26.418 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:06:26.425 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:06:26.428 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:06:27.337 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1275570382 2023-10-17T01:06:27.338 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:06:27.339 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504787 2023-10-17T01:06:27.339 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:06:27.339 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1275570382 ]] 2023-10-17T01:06:27.339 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1275570382 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\4\0\5\6\4\4\4\1\8 ]] 2023-10-17T01:06:27.339 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist add 172.21.15.169:0/1275570382 2023-10-17T01:06:29.147 INFO:tasks.workunit.client.0.smithi169.stderr:blocklisting 172.21.15.169:0/1275570382 until 2023-10-17T02:06:28.364215+0000 (3600 sec) 2023-10-17T01:06:29.182 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist ls 2023-10-17T01:06:29.183 INFO:tasks.workunit.client.0.smithi169.stderr:+ grep -q 172.21.15.169:0/1275570382 2023-10-17T01:06:29.855 INFO:tasks.workunit.client.0.smithi169.stderr:listed 20 entries 2023-10-17T01:06:29.888 INFO:tasks.workunit.client.0.smithi169.stderr:+ PREV_CLIENT_ADDR=172.21.15.169:0/1275570382 2023-10-17T01:06:29.888 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:06:39.892 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:06:39.892 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:06:39.896 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:06:39.905 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:06:40.770 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1275570382 2023-10-17T01:06:40.771 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:06:40.771 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504800 2023-10-17T01:06:40.772 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:06:40.772 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1275570382 ]] 2023-10-17T01:06:40.772 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1275570382 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\2\7\5\5\7\0\3\8\2 ]] 2023-10-17T01:06:40.772 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:06:40.798 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:06:40.796+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:06:40.799 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:06:40.800+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:06:40.803 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:06:40.804+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:06:43.341 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:06:43.340+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:06:43.341 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:06:43.340+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:06:43.341 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:06:43.340+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:06:43.341 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:06:43.340+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:06:43.342 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:06:43.340+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:06:43.720 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:06:43.720+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:06:43.721 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:06:43.724+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:06:43.724 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:06:43.724+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:06:43.724 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:06:43.724+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:06:43.724 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:06:43.724+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:06:43.725 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:06:43.724+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:06:43.725 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:06:43.724+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:06:43.725 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:06:43.724+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:06:43.725 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:06:43.724+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:06:43.725 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:06:43.724+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:06:43.726 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:06:43.724+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:06:43.726 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:06:43.724+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:06:50.773 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:06:50.774 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:06:50.774 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:06:50.776 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:06:51.640 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1275570382 2023-10-17T01:06:51.640 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:06:51.641 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504811 2023-10-17T01:06:51.641 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:06:51.642 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1275570382 ]] 2023-10-17T01:06:51.642 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1275570382 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\2\7\5\5\7\0\3\8\2 ]] 2023-10-17T01:06:51.642 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:07:00.003 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.003+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.004 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.003+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b3b0e000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.006 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.007+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.006 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.007+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4408100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.009 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.011+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.009 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.011+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b5047780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.012 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.011+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.012 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.011+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4408400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.019 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.019+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.019 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.019+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4e53000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.024 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.023+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.024 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.023+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4d44880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.047 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.047+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.048 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.047+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4d45b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.050 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.051+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.050 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.051+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4d45700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.062 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.063+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.062 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.063+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4d45900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.065 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.067+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.066 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.067+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4d45d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.073 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.071+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.073 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.071+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4d45f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.088 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.087+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.089 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.087+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4f3c180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.094 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.091+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.094 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.091+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4f3c380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.096 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.095+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.096 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.095+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4f3c580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.100 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.099+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.100 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.099+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4f3c780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.104 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.103+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.104 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.103+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4f3c980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.108 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.107+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.108 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.107+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4f3d280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.111 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.111+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.112 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.111+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4f3d100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.114 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.115+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.115 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.115+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4f3cf00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.119 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.119+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:00.119 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:07:00.119+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4f3cd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:07:01.643 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:07:01.644 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:07:01.644 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:07:01.647 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:07:02.522 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1275570382 2023-10-17T01:07:02.522 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:07:02.523 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504822 2023-10-17T01:07:02.523 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:07:02.524 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1275570382 ]] 2023-10-17T01:07:02.524 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1275570382 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\2\7\5\5\7\0\3\8\2 ]] 2023-10-17T01:07:02.524 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:07:12.525 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:07:12.530 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:07:12.534 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:07:12.545 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:07:13.564 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1275570382 2023-10-17T01:07:13.564 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:07:13.565 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504833 2023-10-17T01:07:13.565 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:07:13.565 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1275570382 ]] 2023-10-17T01:07:13.565 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1275570382 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\2\7\5\5\7\0\3\8\2 ]] 2023-10-17T01:07:13.566 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:07:16.230 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:07:16.227+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:07:16.233 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:07:16.235+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:07:16.233 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:07:16.235+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbda750 2023-10-17T01:07:16.242 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:07:16.239+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:07:16.242 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:07:16.239+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:07:16.243 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:07:16.239+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:07:16.243 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:07:16.239+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:07:16.243 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:07:16.239+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:07:16.243 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:07:16.239+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:07:16.243 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:07:16.239+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:07:16.244 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:07:16.239+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:07:16.244 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:07:16.239+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:07:16.244 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:07:16.239+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:07:16.244 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:07:16.239+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:07:16.244 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:07:16.239+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:07:23.570 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:07:23.572 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:07:23.574 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:07:23.593 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:07:24.449 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1275570382 2023-10-17T01:07:24.450 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:07:24.450 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504844 2023-10-17T01:07:24.450 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:07:24.450 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1275570382 ]] 2023-10-17T01:07:24.451 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1275570382 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\2\7\5\5\7\0\3\8\2 ]] 2023-10-17T01:07:24.451 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:07:33.629 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:07:33.626+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:07:33.629 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:07:33.626+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:07:34.453 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:07:34.454 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:07:34.457 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:07:34.465 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:07:35.390 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/4244069168 2023-10-17T01:07:35.390 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:07:35.391 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504855 2023-10-17T01:07:35.391 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:07:35.391 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/4244069168 ]] 2023-10-17T01:07:35.391 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/4244069168 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\2\7\5\5\7\0\3\8\2 ]] 2023-10-17T01:07:35.392 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist add 172.21.15.169:0/4244069168 2023-10-17T01:07:37.872 INFO:tasks.workunit.client.0.smithi169.stderr:blocklisting 172.21.15.169:0/4244069168 until 2023-10-17T02:07:37.091296+0000 (3600 sec) 2023-10-17T01:07:37.910 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist ls 2023-10-17T01:07:37.910 INFO:tasks.workunit.client.0.smithi169.stderr:+ grep -q 172.21.15.169:0/4244069168 2023-10-17T01:07:38.717 INFO:tasks.workunit.client.0.smithi169.stderr:listed 21 entries 2023-10-17T01:07:38.745 INFO:tasks.workunit.client.0.smithi169.stderr:+ PREV_CLIENT_ADDR=172.21.15.169:0/4244069168 2023-10-17T01:07:38.745 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:07:48.754 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:07:48.754 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:07:48.754 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:07:48.755 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:07:49.660 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/4244069168 2023-10-17T01:07:49.660 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:07:49.661 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504869 2023-10-17T01:07:49.662 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:07:49.662 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/4244069168 ]] 2023-10-17T01:07:49.662 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/4244069168 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\4\2\4\4\0\6\9\1\6\8 ]] 2023-10-17T01:07:49.662 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:07:56.574 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:07:56.573+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:07:56.574 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:07:56.573+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:07:56.574 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:07:56.573+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:07:56.574 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:07:56.573+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:07:56.575 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:07:56.573+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:07:56.575 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:07:56.573+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:07:56.575 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:07:56.573+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:07:56.575 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:07:56.573+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:07:56.575 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:07:56.573+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:07:56.576 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:07:56.573+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:07:56.576 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:07:56.573+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:07:56.576 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:07:56.573+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:07:56.576 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:07:56.573+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:07:56.576 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:07:56.573+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:07:59.663 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:07:59.664 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:07:59.670 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:07:59.670 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:08:00.003 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.005+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.004 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.005+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b49c5500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.007 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.009+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.007 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.009+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4e52f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.012 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.013+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.012 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.013+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b3af4900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.017 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.017+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.017 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.017+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b3e35100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.022 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.025+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.022 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.025+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4d5c100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.027 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.029+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.027 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.029+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4d33c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.033 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.037+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.033 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.037+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b5046880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.037 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.037+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.038 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.037+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b5047d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.043 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.045+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.043 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.045+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4d44580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.049 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.049+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.049 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.049+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4fee500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.052 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.053+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.052 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.053+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b5047e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.067 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.069+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.067 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.069+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.068 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.069+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b5047180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.068 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.069+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4f3d400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.069 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.069+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.069 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.069+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4384180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.075 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.077+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.075 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.077+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4e16300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.080 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.081+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.080 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.081+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4f3d600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.085 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.085+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.085 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.085+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4f3d800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.089 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.089+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.090 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.089+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4f3da00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.094 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.097+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.095 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.097+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4f3dc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.098 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.101+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.098 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:08:00.101+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4f3de00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:08:00.708 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/4244069168 2023-10-17T01:08:00.714 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:08:00.715 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504880 2023-10-17T01:08:00.715 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:08:00.715 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/4244069168 ]] 2023-10-17T01:08:00.715 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/4244069168 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\4\2\4\4\0\6\9\1\6\8 ]] 2023-10-17T01:08:00.716 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:08:10.718 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:08:10.720 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:08:10.726 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:08:10.742 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:08:11.766 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/4244069168 2023-10-17T01:08:11.766 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:08:11.767 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504891 2023-10-17T01:08:11.767 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:08:11.767 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/4244069168 ]] 2023-10-17T01:08:11.768 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/4244069168 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\4\2\4\4\0\6\9\1\6\8 ]] 2023-10-17T01:08:11.768 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:08:21.771 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:08:21.772 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:08:21.773 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:08:21.775 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:08:22.615 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/4244069168 2023-10-17T01:08:22.615 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:08:22.616 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504902 2023-10-17T01:08:22.616 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:08:22.616 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/4244069168 ]] 2023-10-17T01:08:22.616 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/4244069168 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\4\2\4\4\0\6\9\1\6\8 ]] 2023-10-17T01:08:22.617 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:08:32.618 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:08:32.618 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:08:32.623 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:08:32.624 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:08:33.575 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/4244069168 2023-10-17T01:08:33.575 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:08:33.576 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504913 2023-10-17T01:08:33.576 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:08:33.576 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/4244069168 ]] 2023-10-17T01:08:33.576 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/4244069168 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\4\2\4\4\0\6\9\1\6\8 ]] 2023-10-17T01:08:33.577 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:08:43.582 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:08:43.583 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:08:43.584 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:08:43.597 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:08:44.445 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/3137532849 2023-10-17T01:08:44.445 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:08:44.446 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504924 2023-10-17T01:08:44.446 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:08:44.447 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/3137532849 ]] 2023-10-17T01:08:44.447 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/3137532849 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\4\2\4\4\0\6\9\1\6\8 ]] 2023-10-17T01:08:44.447 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist add 172.21.15.169:0/3137532849 2023-10-17T01:08:46.896 INFO:tasks.workunit.client.0.smithi169.stderr:blocklisting 172.21.15.169:0/3137532849 until 2023-10-17T02:08:46.107318+0000 (3600 sec) 2023-10-17T01:08:46.930 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist ls 2023-10-17T01:08:46.930 INFO:tasks.workunit.client.0.smithi169.stderr:+ grep -q 172.21.15.169:0/3137532849 2023-10-17T01:08:47.698 INFO:tasks.workunit.client.0.smithi169.stderr:listed 22 entries 2023-10-17T01:08:47.729 INFO:tasks.workunit.client.0.smithi169.stderr:+ PREV_CLIENT_ADDR=172.21.15.169:0/3137532849 2023-10-17T01:08:47.729 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:08:57.734 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:08:57.734 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:08:57.735 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:08:57.735 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:08:58.544 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/3137532849 2023-10-17T01:08:58.544 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:08:58.545 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504938 2023-10-17T01:08:58.545 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:08:58.546 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/3137532849 ]] 2023-10-17T01:08:58.546 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/3137532849 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\3\1\3\7\5\3\2\8\4\9 ]] 2023-10-17T01:08:58.546 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:09:00.001 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.000+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.001 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.000+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b5031c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.003 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.004+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.003 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.004+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4d32780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.006 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.008+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.006 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.008+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4dfb800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.022 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.024+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.023 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.024+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4d32380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.024 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.024+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.024 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.024+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4fee580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.028 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.028+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.028 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.028+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4d44f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.044 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.044+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.044 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.044+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4feeb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.049 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.048+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.049 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.048+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4e52100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.053 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.052+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.053 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.052+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4d5c900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.057 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.056+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.058 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.056+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4f3db80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.062 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.064+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.062 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.064+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4f3dd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.066 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.068+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.066 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.068+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b51b8080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.069 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.068+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.070 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.068+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b51b8280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.073 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.072+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.073 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.072+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b51b8480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.076 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.076+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.076 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.076+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b51b8680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.079 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.080+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.080 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.080+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b51b8880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.083 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.084+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.083 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.084+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b51b8a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.088 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.088+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.088 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.088+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b51b8c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.092 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.092+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.092 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.092+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b51b8e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.098 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.096+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:00.098 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:00.096+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b51b9080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:08.547 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:09:08.548 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:09:08.552 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:09:08.557 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:09:09.452 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/3137532849 2023-10-17T01:09:09.452 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:09:09.453 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504949 2023-10-17T01:09:09.453 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:09:09.453 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/3137532849 ]] 2023-10-17T01:09:09.454 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/3137532849 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\3\1\3\7\5\3\2\8\4\9 ]] 2023-10-17T01:09:09.454 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:09:19.459 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:09:19.463 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:09:19.470 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:09:19.480 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:09:20.500 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/3137532849 2023-10-17T01:09:20.500 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:09:20.501 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504960 2023-10-17T01:09:20.501 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:09:20.502 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/3137532849 ]] 2023-10-17T01:09:20.502 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/3137532849 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\3\1\3\7\5\3\2\8\4\9 ]] 2023-10-17T01:09:20.502 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:09:30.509 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:09:30.510 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:09:30.510 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:09:30.510 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:09:31.366 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/3137532849 2023-10-17T01:09:31.367 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:09:31.367 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504971 2023-10-17T01:09:31.368 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:09:31.368 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/3137532849 ]] 2023-10-17T01:09:31.368 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/3137532849 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\3\1\3\7\5\3\2\8\4\9 ]] 2023-10-17T01:09:31.368 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:09:41.371 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:09:41.371 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:09:41.372 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:09:41.393 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:09:42.282 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/3137532849 2023-10-17T01:09:42.284 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:09:42.286 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504982 2023-10-17T01:09:42.286 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:09:42.287 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/3137532849 ]] 2023-10-17T01:09:42.287 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/3137532849 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\3\1\3\7\5\3\2\8\4\9 ]] 2023-10-17T01:09:42.287 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:09:52.289 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:09:52.289 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:09:52.291 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:09:52.309 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:09:53.270 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/551423965 2023-10-17T01:09:53.270 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:09:53.271 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697504993 2023-10-17T01:09:53.271 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:09:53.271 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/551423965 ]] 2023-10-17T01:09:53.271 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/551423965 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\3\1\3\7\5\3\2\8\4\9 ]] 2023-10-17T01:09:53.272 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist add 172.21.15.169:0/551423965 2023-10-17T01:09:55.221 INFO:tasks.workunit.client.0.smithi169.stderr:blocklisting 172.21.15.169:0/551423965 until 2023-10-17T02:09:54.491662+0000 (3600 sec) 2023-10-17T01:09:55.254 INFO:tasks.workunit.client.0.smithi169.stderr:+ grep -q 172.21.15.169:0/551423965 2023-10-17T01:09:55.254 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist ls 2023-10-17T01:09:55.964 INFO:tasks.workunit.client.0.smithi169.stderr:listed 23 entries 2023-10-17T01:09:56.012 INFO:tasks.workunit.client.0.smithi169.stderr:+ PREV_CLIENT_ADDR=172.21.15.169:0/551423965 2023-10-17T01:09:56.012 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:09:59.999 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:59.998+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:09:59.999 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:09:59.998+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b5047e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.002 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.002+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.002 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.002+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4f3d400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.006 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.006+0000 7f34fe0c7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.007 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.006+0000 7f34fe0c7640 -1 librbd::ImageState: 0x55c1b4e16680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.011 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.010+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.011 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.010+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4f3d580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.014 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.014+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.015 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.014+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4d45c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.018 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.014+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.018 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.014+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b51b9300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.020 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.018+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.021 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.018+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b464a600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.028 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.026+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.028 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.026+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b464a480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.037 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.034+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.037 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.034+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b464ab80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.040 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.038+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.040 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.038+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b464aa00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.044 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.042+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.044 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.042+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b464a800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.048 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.046+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.048 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.046+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b51b9180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.052 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.050+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.053 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.050+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b464ae00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.057 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.054+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.057 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.054+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b464b000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.061 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.058+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.061 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.058+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b464b200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.066 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.062+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.066 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.062+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b464b400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.069 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.066+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.069 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.066+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b464b600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.071 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.070+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.072 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.070+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b464b800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.074 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.074+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.074 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.074+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b464ba00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.077 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.074+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:00.077 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:10:00.074+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b464bc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:10:06.013 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:10:06.014 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:10:06.014 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:10:06.015 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:10:06.917 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/551423965 2023-10-17T01:10:06.917 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:10:06.918 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505006 2023-10-17T01:10:06.918 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:10:06.918 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/551423965 ]] 2023-10-17T01:10:06.919 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/551423965 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\5\5\1\4\2\3\9\6\5 ]] 2023-10-17T01:10:06.919 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:10:07.231 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:10:07.226+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:10:07.231 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:10:07.226+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:10:07.231 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:10:07.226+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbda750 2023-10-17T01:10:07.231 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:10:07.226+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:10:07.232 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:10:07.226+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:10:07.232 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:10:07.226+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:10:07.232 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:10:07.226+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:10:07.232 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:10:07.226+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:10:07.232 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:10:07.226+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:10:07.232 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:10:07.226+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:10:07.233 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:10:07.226+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:10:07.233 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:10:07.226+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:10:07.233 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:10:07.226+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:10:07.233 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:10:07.226+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:10:07.234 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:10:07.226+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:10:12.748 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:10:12.742+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:10:12.748 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:10:12.742+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:10:12.749 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:10:12.742+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:10:12.754 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:10:12.758+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:10:16.920 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:10:16.920 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:10:16.930 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:10:16.950 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:10:17.887 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/551423965 2023-10-17T01:10:17.887 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:10:17.888 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505017 2023-10-17T01:10:17.888 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:10:17.888 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/551423965 ]] 2023-10-17T01:10:17.888 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/551423965 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\5\5\1\4\2\3\9\6\5 ]] 2023-10-17T01:10:17.889 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:10:27.890 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:10:27.891 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:10:27.892 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:10:27.902 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:10:28.899 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/551423965 2023-10-17T01:10:28.899 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:10:28.900 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505028 2023-10-17T01:10:28.900 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:10:28.900 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/551423965 ]] 2023-10-17T01:10:28.901 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/551423965 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\5\5\1\4\2\3\9\6\5 ]] 2023-10-17T01:10:28.901 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:10:34.886 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:10:34.886+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:10:34.889 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:10:34.890+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:10:34.889 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:10:34.890+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:10:34.889 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:10:34.890+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:10:34.889 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:10:34.890+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:10:34.890 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:10:34.894+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:10:38.903 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:10:38.910 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:10:38.919 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:10:38.938 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:10:39.962 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/551423965 2023-10-17T01:10:39.965 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:10:39.966 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505039 2023-10-17T01:10:39.966 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:10:39.966 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/551423965 ]] 2023-10-17T01:10:39.966 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/551423965 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\5\5\1\4\2\3\9\6\5 ]] 2023-10-17T01:10:39.967 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:10:41.842 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:10:41.841+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:10:41.843 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:10:41.841+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:10:41.843 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:10:41.841+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:10:41.843 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:10:41.841+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:10:49.122 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:10:49.121+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:10:49.123 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:10:49.121+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:10:49.123 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:10:49.121+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:10:49.123 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:10:49.121+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:10:49.971 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:10:49.974 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:10:49.976 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:10:49.980 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:10:50.938 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/551423965 2023-10-17T01:10:50.939 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:10:50.941 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505050 2023-10-17T01:10:50.941 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:10:50.941 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/551423965 ]] 2023-10-17T01:10:50.941 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/551423965 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\5\5\1\4\2\3\9\6\5 ]] 2023-10-17T01:10:50.942 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:11:00.953 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:11:00.957 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:11:00.960 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:11:00.964 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:11:02.161 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/551423965 2023-10-17T01:11:02.167 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:11:02.168 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505062 2023-10-17T01:11:02.169 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:11:02.169 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/551423965 ]] 2023-10-17T01:11:02.169 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/551423965 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\5\5\1\4\2\3\9\6\5 ]] 2023-10-17T01:11:02.169 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:11:04.113 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:11:04.113+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:11:04.113 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:11:04.113+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:11:04.117 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:11:04.117+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbda750 2023-10-17T01:11:04.117 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:11:04.117+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:11:04.117 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:11:04.117+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:11:04.117 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:11:04.117+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:11:04.117 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:11:04.117+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:11:04.117 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:11:04.117+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:11:04.118 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:11:04.117+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:11:04.118 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:11:04.117+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:11:04.118 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:11:04.117+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:11:04.118 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:11:04.117+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:11:04.118 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:11:04.117+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:11:04.119 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:11:04.117+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:11:04.119 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:11:04.117+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:11:12.170 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:11:12.171 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:11:12.171 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:11:12.182 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:11:13.102 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:11:13.102 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:11:13.103 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505073 2023-10-17T01:11:13.103 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:11:13.103 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:11:13.103 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\5\5\1\4\2\3\9\6\5 ]] 2023-10-17T01:11:13.103 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist add 172.21.15.169:0/1398705567 2023-10-17T01:11:14.926 INFO:tasks.workunit.client.0.smithi169.stderr:blocklisting 172.21.15.169:0/1398705567 until 2023-10-17T02:11:14.162954+0000 (3600 sec) 2023-10-17T01:11:14.953 INFO:tasks.workunit.client.0.smithi169.stderr:+ ceph osd blocklist ls 2023-10-17T01:11:14.962 INFO:tasks.workunit.client.0.smithi169.stderr:+ grep -q 172.21.15.169:0/1398705567 2023-10-17T01:11:15.691 INFO:tasks.workunit.client.0.smithi169.stderr:listed 24 entries 2023-10-17T01:11:15.722 INFO:tasks.workunit.client.0.smithi169.stderr:+ PREV_CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:11:15.722 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:11:25.725 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:11:25.726 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:11:25.728 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:11:25.744 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:11:26.741 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:11:26.742 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:11:26.744 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505086 2023-10-17T01:11:26.744 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:11:26.744 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:11:26.744 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:11:26.745 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:11:36.752 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:11:36.753 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:11:36.754 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:11:36.766 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:11:37.574 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:11:37.575 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:11:37.575 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505097 2023-10-17T01:11:37.575 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:11:37.576 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:11:37.576 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:11:37.576 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:11:47.584 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:11:47.585 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:11:47.585 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:11:47.587 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:11:48.510 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:11:48.521 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:11:48.522 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505108 2023-10-17T01:11:48.523 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:11:48.523 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:11:48.523 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:11:48.523 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:11:55.904 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:11:55.900+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:11:55.911 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:11:55.904+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:11:55.911 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:11:55.904+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:11:55.911 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:11:55.904+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:11:58.524 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:11:58.525 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:11:58.525 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:11:58.526 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:11:59.363 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:11:59.363 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:11:59.364 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505119 2023-10-17T01:11:59.364 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:11:59.365 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:11:59.365 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:11:59.365 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:12:00.003 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:12:00.003+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:12:00.003 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:12:00.003+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b431f800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:12:00.004 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:12:00.003+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:12:00.004 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:12:00.003+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b5047400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:12:00.006 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:12:00.003+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:12:00.006 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:12:00.003+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b51b8c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:12:00.010 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:12:00.011+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:12:00.011 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:12:00.011+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b5047200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:12:00.013 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:12:00.011+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:12:00.014 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:12:00.011+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b51b9980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:12:00.018 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:12:00.015+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:12:00.018 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:12:00.015+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b51b9b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:12:00.022 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:12:00.019+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:12:00.022 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:12:00.019+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b51b8a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:12:00.026 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:12:00.023+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:12:00.026 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:12:00.023+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b51b9600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:12:00.039 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:12:00.039+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:12:00.039 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:12:00.039+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:12:00.039 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:12:00.039+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4f3c800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:12:00.040 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:12:00.039+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4bcc300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:12:00.059 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:12:00.059+0000 7f34fd8c6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:12:00.059 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:12:00.059+0000 7f34fd8c6640 -1 librbd::ImageState: 0x55c1b4feeb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:12:00.067 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:12:00.067+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:12:00.068 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:12:00.067+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4a3fb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:12:00.071 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:12:00.071+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:12:00.071 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:12:00.071+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b464af00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:12:00.074 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:12:00.075+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:12:00.075 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:12:00.075+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4e53c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:12:00.078 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:12:00.079+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:12:00.079 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:12:00.079+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b4f3d800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:12:00.082 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:12:00.079+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:12:00.082 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:12:00.083+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b464bd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:12:00.086 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:12:00.083+0000 7f34fd0c5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:12:00.086 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:12:00.083+0000 7f34fd0c5640 -1 librbd::ImageState: 0x55c1b521c100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-17T01:12:07.735 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:12:07.731+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:12:07.735 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:12:07.731+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:12:07.735 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:12:07.731+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbda750 2023-10-17T01:12:07.737 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:12:07.731+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:12:07.737 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:12:07.731+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:12:07.737 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:12:07.731+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:12:07.737 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:12:07.731+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:12:07.737 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:12:07.731+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:12:07.738 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:12:07.731+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:12:07.738 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:12:07.731+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:12:07.738 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:12:07.731+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:12:07.738 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:12:07.731+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:12:07.738 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:12:07.731+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:12:07.739 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:12:07.731+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:12:07.739 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:12:07.731+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:12:09.377 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:12:09.377 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:12:09.382 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:12:09.412 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:12:10.288 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:12:10.289 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:12:10.289 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505130 2023-10-17T01:12:10.290 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:12:10.290 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:12:10.290 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:12:10.290 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:12:20.291 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:12:20.292 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:12:20.296 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:12:20.298 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:12:21.101 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:12:21.102 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:12:21.102 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505141 2023-10-17T01:12:21.102 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:12:21.103 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:12:21.103 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:12:21.103 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:12:31.105 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:12:31.105 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:12:31.109 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:12:31.132 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:12:31.938 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:12:31.938 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:12:31.939 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505151 2023-10-17T01:12:31.940 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:12:31.940 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:12:31.940 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:12:31.940 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:12:41.947 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:12:41.948 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:12:41.949 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:12:41.966 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:12:42.812 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:12:42.812 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:12:42.813 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505162 2023-10-17T01:12:42.813 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:12:42.813 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:12:42.813 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:12:42.814 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:12:52.821 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:12:52.825 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:12:52.830 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:12:52.839 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:12:53.754 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:12:53.754 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:12:53.755 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505173 2023-10-17T01:12:53.755 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:12:53.756 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:12:53.756 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:12:53.756 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:13:03.757 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:13:03.757 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:13:03.758 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:13:03.758 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:13:04.720 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:13:04.720 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:13:04.724 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505184 2023-10-17T01:13:04.724 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:13:04.724 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:13:04.724 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:13:04.724 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:13:14.728 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:13:14.730 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:13:14.732 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:13:14.742 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:13:15.624 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:13:15.624 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:13:15.625 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505195 2023-10-17T01:13:15.625 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:13:15.625 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:13:15.625 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:13:15.625 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:13:22.894 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:13:22.893+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:13:22.900 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:13:22.893+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:13:22.900 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:13:22.893+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:13:22.910 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:13:22.905+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:13:22.910 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:13:22.905+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:13:22.910 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:13:22.905+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:13:25.630 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:13:25.630 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:13:25.637 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:13:25.644 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:13:26.625 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:13:26.625 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:13:26.626 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505206 2023-10-17T01:13:26.626 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:13:26.627 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:13:26.627 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:13:26.627 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:13:36.628 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:13:36.638 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:13:36.649 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:13:36.652 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:13:37.602 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:13:37.602 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:13:37.603 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505217 2023-10-17T01:13:37.603 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:13:37.603 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:13:37.604 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:13:37.604 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:13:47.611 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:13:47.616 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:13:47.619 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:13:47.628 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:13:47.818 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:13:47.816+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:13:47.818 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:13:47.816+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:13:47.821 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:13:47.820+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:13:47.821 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:13:47.820+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:13:47.821 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:13:47.820+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:13:47.822 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:13:47.820+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:13:47.822 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:13:47.820+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:13:47.822 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:13:47.820+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:13:47.822 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:13:47.820+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:13:47.822 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:13:47.820+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:13:47.823 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:13:47.820+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:13:47.823 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:13:47.820+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:13:47.823 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:13:47.820+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:13:47.823 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:13:47.820+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:13:48.485 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:13:48.485 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:13:48.486 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505228 2023-10-17T01:13:48.486 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:13:48.486 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:13:48.487 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:13:48.487 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:13:58.488 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:13:58.488 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:13:58.489 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:13:58.489 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:13:59.419 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:13:59.441 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:13:59.442 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505239 2023-10-17T01:13:59.442 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:13:59.442 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:13:59.443 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:13:59.443 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:14:09.423 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:14:09.423 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:14:09.424 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:14:09.432 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:14:10.317 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:14:10.319 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:14:10.321 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505250 2023-10-17T01:14:10.321 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:14:10.321 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:14:10.321 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:14:10.321 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:14:20.341 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:14:20.341 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:14:20.352 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:14:20.355 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:14:21.291 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:14:21.304 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:14:21.305 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505261 2023-10-17T01:14:21.305 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:14:21.306 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:14:21.306 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:14:21.306 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:14:29.140 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:14:29.131+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:14:31.310 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:14:31.314 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:14:31.317 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:14:31.322 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:14:32.270 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:14:32.273 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:14:32.274 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505272 2023-10-17T01:14:32.274 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:14:32.275 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:14:32.275 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:14:32.275 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:14:42.290 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:14:42.295 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:14:42.298 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:14:42.300 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:14:43.179 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:14:43.179 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:14:43.180 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505283 2023-10-17T01:14:43.180 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:14:43.180 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:14:43.181 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:14:43.181 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:14:53.182 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:14:53.182 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:14:53.183 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:14:53.184 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:14:54.120 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:14:54.120 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:14:54.121 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505294 2023-10-17T01:14:54.121 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:14:54.122 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:14:54.122 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:14:54.122 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:15:04.125 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:15:04.125 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:15:04.125 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:15:04.125 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:15:05.088 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:15:05.088 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:15:05.089 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505305 2023-10-17T01:15:05.089 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:15:05.089 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:15:05.090 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:15:05.090 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:15:05.562 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:05.561+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:15:05.562 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:05.561+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:15:05.562 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:05.561+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:15:05.562 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:05.561+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:15:15.102 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:15:15.103 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:15:15.103 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:15:15.103 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:15:16.053 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:15:16.053 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:15:16.054 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505316 2023-10-17T01:15:16.054 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:15:16.055 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:15:16.055 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:15:16.055 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:15:24.220 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:24.217+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:15:24.220 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:24.217+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:15:24.228 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:24.225+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbda750 2023-10-17T01:15:24.228 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:24.225+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:15:24.228 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:24.225+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:15:24.228 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:24.225+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:15:24.229 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:24.225+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:15:24.229 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:24.225+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:15:24.229 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:24.225+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:15:24.229 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:24.225+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:15:24.229 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:24.225+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:15:24.230 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:24.225+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:15:24.230 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:24.225+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:15:24.230 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:24.225+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:15:24.230 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:24.225+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:15:26.062 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:15:26.063 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:15:26.080 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:15:26.091 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:15:26.982 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:15:26.983 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:15:26.983 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505326 2023-10-17T01:15:26.983 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:15:26.983 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:15:26.984 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:15:26.984 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:15:36.985 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:15:36.991 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:15:36.997 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:15:37.001 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:15:37.866 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:15:37.867 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:15:37.867 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505337 2023-10-17T01:15:37.867 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:15:37.868 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:15:37.868 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:15:37.868 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:15:47.869 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:15:47.873 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:15:47.874 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:15:47.882 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:15:48.735 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:48.732+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:15:48.735 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:48.732+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:15:48.735 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:48.732+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:15:48.740 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:48.736+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:15:48.740 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:48.736+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:15:48.740 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:48.736+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:15:48.741 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:48.736+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:15:48.744 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:48.740+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:15:48.744 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:48.740+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:15:48.744 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:48.740+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:15:48.861 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:48.856+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:15:48.861 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:48.856+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:15:48.861 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:48.856+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbda750 2023-10-17T01:15:48.862 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:48.856+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:15:48.862 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:48.856+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:15:48.862 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:48.856+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:15:48.862 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:48.856+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:15:48.862 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:48.856+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:15:48.863 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:48.856+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:15:48.863 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:48.856+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:15:48.863 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:48.856+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:15:48.863 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:48.856+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:15:48.863 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:48.856+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:15:48.864 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:48.856+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:15:48.864 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:15:48.856+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:15:48.989 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:15:48.989 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:15:48.990 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505348 2023-10-17T01:15:48.990 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:15:48.991 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:15:48.991 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:15:48.991 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:15:58.992 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:15:58.997 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:15:58.998 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:15:59.013 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:15:59.949 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:15:59.949 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:15:59.950 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505359 2023-10-17T01:15:59.950 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:15:59.950 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:15:59.950 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:15:59.951 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:16:09.950 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:16:09.954 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:16:09.965 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:16:09.969 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:16:10.818 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:16:10.823 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:16:10.824 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505370 2023-10-17T01:16:10.824 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:16:10.825 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:16:10.825 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:16:10.825 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:16:20.839 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:16:20.839 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:16:20.841 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:16:20.844 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:16:21.706 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:16:21.713 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:16:21.715 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505381 2023-10-17T01:16:21.715 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:16:21.715 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:16:21.716 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:16:21.716 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:16:31.722 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:16:31.723 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:16:31.727 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:16:31.729 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:16:32.715 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:16:32.715 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:16:32.716 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505392 2023-10-17T01:16:32.716 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:16:32.716 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:16:32.717 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:16:32.717 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:16:42.727 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:16:42.727 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:16:42.744 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:16:42.746 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:16:43.603 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:16:43.603 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:16:43.603 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505403 2023-10-17T01:16:43.604 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:16:43.604 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:16:43.604 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:16:43.604 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:16:53.606 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:16:53.606 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:16:53.612 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:16:53.620 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:16:54.593 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:16:54.594 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:16:54.594 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505414 2023-10-17T01:16:54.595 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:16:54.595 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:16:54.595 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:16:54.595 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:17:03.539 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:17:03.530+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:17:03.539 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:17:03.530+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:17:03.539 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:17:03.530+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:17:04.611 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:17:04.611 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:17:04.616 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:17:04.618 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:17:05.604 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:17:05.604 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:17:05.605 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505425 2023-10-17T01:17:05.605 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:17:05.605 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:17:05.606 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:17:05.606 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:17:15.607 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:17:15.609 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:17:15.620 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:17:15.626 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:17:16.631 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:17:16.631 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:17:16.632 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505436 2023-10-17T01:17:16.632 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:17:16.632 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:17:16.632 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:17:16.632 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:17:26.642 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:17:26.643 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:17:26.647 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:17:26.654 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:17:27.690 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:17:27.690 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:17:27.691 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505447 2023-10-17T01:17:27.691 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:17:27.692 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:17:27.692 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:17:27.692 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:17:37.692 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:17:37.693 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:17:37.694 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:17:37.694 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:17:38.525 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:17:38.526 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:17:38.526 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505458 2023-10-17T01:17:38.527 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:17:38.527 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:17:38.527 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:17:38.527 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:17:48.528 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:17:48.529 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:17:48.536 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:17:48.545 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:17:49.393 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:17:49.394 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:17:49.394 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505469 2023-10-17T01:17:49.394 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:17:49.395 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:17:49.395 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:17:49.395 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:17:59.398 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:17:59.402 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:17:59.402 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:17:59.403 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:18:00.233 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:18:00.234 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:18:00.234 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505480 2023-10-17T01:18:00.234 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:18:00.235 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:18:00.235 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:18:00.235 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:18:10.241 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:18:10.242 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:18:10.246 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:18:10.262 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:18:11.094 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:18:11.095 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:18:11.095 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505491 2023-10-17T01:18:11.096 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:18:11.096 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:18:11.096 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:18:11.096 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:18:21.102 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:18:21.103 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:18:21.106 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:18:21.112 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:18:22.085 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:18:22.085 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:18:22.085 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505502 2023-10-17T01:18:22.086 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:18:22.086 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:18:22.086 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:18:22.086 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:18:22.657 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:18:22.652+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:18:22.661 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:18:22.656+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:18:22.665 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:18:22.660+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:18:22.726 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:18:22.720+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:18:22.726 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:18:22.720+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:18:32.087 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:18:32.088 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:18:32.098 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:18:32.101 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:18:32.974 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:18:32.975 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:18:32.975 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505512 2023-10-17T01:18:32.975 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:18:32.976 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:18:32.976 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:18:32.976 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:18:42.981 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:18:42.983 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:18:43.002 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:18:43.005 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:18:43.880 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:18:43.880 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:18:43.881 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505523 2023-10-17T01:18:43.881 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:18:43.881 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:18:43.881 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:18:43.882 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:18:51.043 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:18:51.039+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:18:51.044 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:18:51.039+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:18:51.053 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:18:51.047+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:18:53.883 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:18:53.883 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:18:53.883 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:18:53.891 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:18:54.789 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:18:54.789 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:18:54.793 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505534 2023-10-17T01:18:54.793 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:18:54.793 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:18:54.794 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:18:54.794 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:18:55.660 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:18:55.647+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:18:55.660 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:18:55.647+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:18:55.660 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:18:55.655+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:18:55.665 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:18:55.655+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:18:55.666 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:18:55.655+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:18:55.666 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:18:55.655+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:18:55.666 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:18:55.655+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:18:55.667 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:18:55.659+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:18:55.667 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:18:55.659+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:18:55.667 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:18:55.659+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:18:55.667 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:18:55.659+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:18:55.667 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:18:55.659+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:18:55.668 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:18:55.659+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:18:55.668 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:18:55.659+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:18:58.391 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:18:58.387+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:18:58.391 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:18:58.387+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:18:58.392 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:18:58.387+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:19:04.797 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:19:04.797 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:19:04.806 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:19:04.822 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:19:05.707 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:19:05.708 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:19:05.708 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505545 2023-10-17T01:19:05.708 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:19:05.708 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:19:05.709 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:19:05.709 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:19:08.666 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:19:08.655+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:19:08.666 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:19:08.659+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:19:08.669 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:19:08.663+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:19:08.669 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:19:08.663+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:19:08.669 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:19:08.663+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:19:08.669 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:19:08.663+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:19:08.669 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:19:08.663+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:19:08.670 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:19:08.663+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:19:08.670 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:19:08.663+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:19:08.670 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:19:08.663+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:19:08.670 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:19:08.663+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:19:08.670 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:19:08.663+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:19:08.671 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:19:08.663+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:19:08.671 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:19:08.663+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:19:15.723 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:19:15.725 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:19:15.725 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:19:15.733 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:19:16.644 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:19:16.658 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:19:16.660 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505556 2023-10-17T01:19:16.661 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:19:16.661 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:19:16.661 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:19:16.661 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:19:26.662 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:19:26.663 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:19:26.663 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:19:26.687 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:19:27.688 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:19:27.688 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:19:27.689 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505567 2023-10-17T01:19:27.689 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:19:27.689 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:19:27.690 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:19:27.690 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:19:37.695 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:19:37.696 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:19:37.698 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:19:37.701 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:19:38.620 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:19:38.621 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:19:38.621 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505578 2023-10-17T01:19:38.621 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:19:38.622 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:19:38.622 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:19:38.622 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:19:48.625 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:19:48.628 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:19:48.628 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:19:48.630 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:19:49.602 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:19:49.606 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:19:49.607 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505589 2023-10-17T01:19:49.607 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:19:49.608 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:19:49.608 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:19:49.608 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:19:59.609 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:19:59.610 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:19:59.614 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:19:59.624 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:20:00.541 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:20:00.546 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:20:00.548 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505600 2023-10-17T01:20:00.548 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:20:00.548 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:20:00.549 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:20:00.549 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:20:10.315 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:20:10.310+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:20:10.316 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:20:10.314+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:20:10.317 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:20:10.314+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:20:10.317 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:20:10.314+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:20:10.317 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:20:10.314+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:20:10.336 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:20:10.334+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:20:10.337 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:20:10.334+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:20:10.337 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:20:10.334+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:20:10.337 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:20:10.334+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:20:10.337 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:20:10.334+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:20:10.337 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:20:10.334+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:20:10.562 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:20:10.567 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:20:10.568 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:20:10.575 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:20:11.382 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:20:11.382 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:20:11.383 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505611 2023-10-17T01:20:11.383 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:20:11.383 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:20:11.383 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:20:11.383 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:20:20.754 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:20:20.749+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:20:20.754 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:20:20.749+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:20:20.754 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:20:20.749+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbda750 2023-10-17T01:20:20.755 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:20:20.749+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:20:20.755 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:20:20.749+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:20:20.755 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:20:20.749+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:20:20.755 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:20:20.749+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:20:20.755 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:20:20.749+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:20:20.756 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:20:20.749+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:20:20.756 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:20:20.749+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:20:20.756 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:20:20.749+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:20:20.756 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:20:20.749+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:20:20.756 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:20:20.749+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:20:20.757 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:20:20.749+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:20:20.757 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:20:20.749+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:20:21.389 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:20:21.389 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:20:21.389 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:20:21.401 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:20:22.255 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:20:22.255 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:20:22.256 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505622 2023-10-17T01:20:22.256 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:20:22.257 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:20:22.257 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:20:22.257 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:20:32.258 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:20:32.259 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:20:32.261 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:20:32.271 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:20:33.254 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:20:33.254 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:20:33.255 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505633 2023-10-17T01:20:33.255 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:20:33.255 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:20:33.255 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:20:33.256 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:20:43.257 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:20:43.257 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:20:43.258 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:20:43.275 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:20:44.255 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:20:44.256 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:20:44.256 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505644 2023-10-17T01:20:44.256 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:20:44.256 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:20:44.257 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:20:44.257 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:20:46.574 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:20:46.553+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:20:46.575 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:20:46.557+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:20:46.579 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:20:46.565+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:20:46.580 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:20:46.577+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:20:54.258 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:20:54.259 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:20:54.259 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:20:54.262 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:20:55.248 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:20:55.248 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:20:55.249 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505655 2023-10-17T01:20:55.249 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:20:55.250 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:20:55.250 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:20:55.250 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:21:05.253 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:21:05.256 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:21:05.260 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:21:05.260 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:21:06.185 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:21:06.185 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:21:06.185 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505666 2023-10-17T01:21:06.186 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:21:06.186 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:21:06.186 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:21:06.186 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:21:16.187 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:21:16.190 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:21:16.202 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:21:16.203 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:21:17.205 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:21:17.205 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:21:17.206 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505677 2023-10-17T01:21:17.206 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:21:17.206 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:21:17.207 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:21:17.207 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:21:27.211 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:21:27.212 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:21:27.214 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:21:27.217 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:21:28.113 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:21:28.113 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:21:28.113 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505688 2023-10-17T01:21:28.114 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:21:28.114 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:21:28.114 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:21:28.114 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:21:38.121 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:21:38.121 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:21:38.121 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:21:38.122 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:21:39.096 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:21:39.096 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:21:39.097 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505699 2023-10-17T01:21:39.097 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:21:39.097 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:21:39.097 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:21:39.098 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:21:49.104 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:21:49.105 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:21:49.111 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:21:49.113 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:21:50.075 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:21:50.076 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:21:50.076 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505710 2023-10-17T01:21:50.077 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:21:50.077 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:21:50.077 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:21:50.077 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:21:54.703 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:21:54.700+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:21:54.708 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:21:54.704+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:21:54.708 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:21:54.704+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:22:00.078 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:22:00.079 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:22:00.087 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:22:00.110 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:22:01.072 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:22:01.073 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:22:01.080 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505721 2023-10-17T01:22:01.080 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:22:01.080 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:22:01.081 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:22:01.081 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:22:01.375 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:01.371+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:22:01.387 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:01.379+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:22:01.387 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:01.379+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbda750 2023-10-17T01:22:01.387 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:01.379+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:22:01.388 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:01.379+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:22:01.388 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:01.379+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:22:01.388 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:01.379+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:22:01.391 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:01.387+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:22:01.391 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:01.387+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:22:01.391 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:01.387+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:22:01.392 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:01.387+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:22:01.392 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:01.387+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:22:01.393 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:01.387+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:22:01.393 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:01.387+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:22:01.394 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:01.387+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:22:11.084 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:22:11.090 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:22:11.093 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:22:11.100 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:22:11.988 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:22:11.989 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:22:11.990 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505731 2023-10-17T01:22:11.990 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:22:11.990 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:22:11.990 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:22:11.991 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:22:16.925 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:16.919+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:22:16.925 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:16.919+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:22:16.934 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:16.927+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:22:16.936 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:16.931+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:22:16.936 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:16.931+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:22:16.937 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:16.931+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:22:16.937 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:16.931+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:22:16.940 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:16.935+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:22:16.940 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:16.935+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:22:16.940 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:16.935+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:22:16.940 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:16.935+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:22:16.941 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:16.935+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:22:16.941 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:16.935+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:22:16.941 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:16.935+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:22:21.997 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:22:21.998 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:22:22.001 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:22:22.014 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:22:22.927 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:22:22.931 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:22:22.932 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505742 2023-10-17T01:22:22.932 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:22:22.932 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:22:22.932 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:22:22.933 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:22:32.945 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:22:32.948 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:22:32.951 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:22:32.955 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:22:33.834 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:22:33.834 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:22:33.834 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505753 2023-10-17T01:22:33.834 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:22:33.835 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:22:33.835 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:22:33.835 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:22:43.835 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:22:43.836 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:22:43.836 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:22:43.837 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:22:44.739 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:22:44.739 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:22:44.740 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505764 2023-10-17T01:22:44.740 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:22:44.740 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:22:44.740 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:22:44.740 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:22:51.718 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:51.710+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:22:51.718 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:51.710+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:22:51.725 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:51.718+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:22:51.725 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:51.718+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:22:51.737 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:51.726+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:22:51.737 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:51.726+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:22:51.737 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:51.726+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:22:51.738 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:22:51.726+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:22:54.744 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:22:54.745 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:22:54.745 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:22:54.759 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:22:55.755 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:22:55.755 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:22:55.756 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505775 2023-10-17T01:22:55.756 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:22:55.756 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:22:55.756 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:22:55.757 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:23:01.863 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:23:01.846+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:23:01.863 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:23:01.846+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:23:05.760 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:23:05.762 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:23:05.763 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:23:05.773 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:23:06.754 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:23:06.755 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:23:06.755 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505786 2023-10-17T01:23:06.756 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:23:06.756 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:23:06.756 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:23:06.756 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:23:07.639 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:23:07.634+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:23:07.639 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:23:07.634+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:23:07.639 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:23:07.634+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:23:07.639 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:23:07.634+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:23:07.646 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:23:07.642+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:23:07.648 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:23:07.646+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:23:07.649 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:23:07.646+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:23:07.649 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:23:07.646+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:23:07.649 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:23:07.646+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:23:07.649 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:23:07.646+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:23:07.649 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:23:07.646+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:23:16.757 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:23:16.758 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:23:16.765 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:23:16.771 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:23:17.670 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:23:17.670 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:23:17.671 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505797 2023-10-17T01:23:17.671 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:23:17.671 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:23:17.671 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:23:17.672 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:23:19.090 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:23:19.086+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:23:25.993 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:23:25.978+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:23:25.993 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:23:25.986+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:23:25.993 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:23:25.986+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:23:25.994 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:23:25.990+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:23:27.683 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:23:27.684 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:23:27.684 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:23:27.686 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:23:28.488 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:23:28.489 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:23:28.489 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505808 2023-10-17T01:23:28.489 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:23:28.490 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:23:28.490 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:23:28.490 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:23:38.500 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:23:38.501 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:23:38.503 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:23:38.511 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:23:39.524 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:23:39.524 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:23:39.525 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505819 2023-10-17T01:23:39.525 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:23:39.525 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:23:39.525 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:23:39.526 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:23:49.527 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:23:49.530 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:23:49.538 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:23:49.544 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:23:50.645 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:23:50.646 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:23:50.646 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505830 2023-10-17T01:23:50.647 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:23:50.647 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:23:50.647 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:23:50.647 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:24:00.651 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:24:00.651 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:24:00.662 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:24:00.678 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:24:01.738 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:24:01.739 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:24:01.739 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505841 2023-10-17T01:24:01.740 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:24:01.740 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:24:01.740 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:24:01.740 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:24:11.741 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:24:11.741 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:24:11.753 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:24:11.755 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:24:12.630 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:24:12.630 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:24:12.638 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505852 2023-10-17T01:24:12.639 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:24:12.639 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:24:12.639 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:24:12.639 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:24:22.643 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:24:22.649 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:24:22.651 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:24:22.655 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:24:23.534 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:24:23.535 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:24:23.535 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505863 2023-10-17T01:24:23.535 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:24:23.535 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:24:23.536 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:24:23.536 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:24:33.545 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:24:33.546 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:24:33.546 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:24:33.558 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:24:34.462 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:24:34.463 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:24:34.464 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505874 2023-10-17T01:24:34.464 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:24:34.464 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:24:34.465 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:24:34.465 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:24:44.471 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:24:44.473 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:24:44.474 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:24:44.489 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:24:45.287 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:24:45.287 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:24:45.287 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505885 2023-10-17T01:24:45.288 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:24:45.288 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:24:45.288 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:24:45.288 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:24:45.319 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:24:45.316+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:24:45.319 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:24:45.316+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:24:55.292 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:24:55.294 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:24:55.295 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:24:55.295 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:24:56.238 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:24:56.239 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:24:56.239 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505896 2023-10-17T01:24:56.239 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:24:56.239 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:24:56.240 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:24:56.240 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:25:06.242 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:25:06.242 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:25:06.254 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:25:06.258 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:25:07.062 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:25:07.063 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:25:07.063 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505907 2023-10-17T01:25:07.063 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:25:07.064 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:25:07.064 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:25:07.064 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:25:16.043 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:25:16.032+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:25:16.047 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:25:16.032+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:25:16.048 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:25:16.044+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:25:16.048 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:25:16.044+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:25:16.054 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:25:16.048+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:25:16.055 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:25:16.048+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:25:16.055 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:25:16.048+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:25:16.055 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:25:16.048+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:25:16.055 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:25:16.048+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:25:17.066 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:25:17.067 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:25:17.067 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:25:17.069 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:25:18.068 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:25:18.068 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:25:18.069 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505918 2023-10-17T01:25:18.069 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:25:18.069 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:25:18.069 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:25:18.070 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:25:28.075 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:25:28.076 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:25:28.076 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:25:28.079 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:25:28.978 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:25:28.983 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:25:28.985 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505928 2023-10-17T01:25:28.985 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:25:28.985 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:25:28.985 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:25:28.985 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:25:38.990 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:25:38.991 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:25:38.991 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:25:38.999 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:25:39.903 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:25:39.903 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:25:39.904 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505939 2023-10-17T01:25:39.904 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:25:39.904 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:25:39.905 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:25:39.905 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:25:49.906 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:25:49.907 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:25:49.907 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:25:49.914 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:25:50.898 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:25:50.898 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:25:50.899 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505950 2023-10-17T01:25:50.899 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:25:50.899 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:25:50.899 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:25:50.900 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:25:57.943 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:25:57.935+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:25:57.944 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:25:57.935+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:25:57.944 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:25:57.935+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbda750 2023-10-17T01:25:57.944 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:25:57.935+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:25:57.944 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:25:57.935+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:25:57.944 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:25:57.935+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:25:57.945 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:25:57.935+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:25:57.945 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:25:57.935+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:25:57.945 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:25:57.935+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:25:57.945 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:25:57.935+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:25:57.945 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:25:57.935+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:25:57.946 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:25:57.935+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:25:57.946 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:25:57.935+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:25:57.946 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:25:57.935+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:25:57.946 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:25:57.935+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:26:00.900 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:26:00.900 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:26:00.901 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:26:00.901 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:26:01.783 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:26:01.783 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:26:01.784 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505961 2023-10-17T01:26:01.784 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:26:01.784 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:26:01.785 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:26:01.785 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:26:06.617 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:26:06.611+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:26:06.641 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:26:06.631+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:26:11.786 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:26:11.786 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:26:11.787 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:26:11.802 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:26:12.681 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:26:12.685 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:26:12.686 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505972 2023-10-17T01:26:12.687 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:26:12.687 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:26:12.687 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:26:12.687 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:26:19.526 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:26:19.519+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:26:19.526 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:26:19.519+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:26:19.526 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:26:19.519+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:26:19.526 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:26:19.519+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:26:22.310 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:26:22.303+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:26:22.311 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:26:22.303+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:26:22.311 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:26:22.303+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:26:22.314 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:26:22.311+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:26:22.322 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:26:22.319+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:26:22.692 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:26:22.693 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:26:22.693 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:26:22.698 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:26:23.777 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:26:23.777 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:26:23.778 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505983 2023-10-17T01:26:23.778 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:26:23.778 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:26:23.778 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:26:23.779 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:26:33.785 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:26:33.790 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:26:33.796 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:26:33.796 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:26:34.679 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:26:34.679 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:26:34.680 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697505994 2023-10-17T01:26:34.680 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:26:34.680 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:26:34.681 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:26:34.681 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:26:44.682 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:26:44.682 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:26:44.689 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:26:44.704 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:26:45.578 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:26:45.578 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:26:45.579 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506005 2023-10-17T01:26:45.579 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:26:45.579 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:26:45.579 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:26:45.579 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:26:55.582 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:26:55.582 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:26:55.583 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:26:55.586 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:26:56.649 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:26:56.650 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:26:56.651 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506016 2023-10-17T01:26:56.651 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:26:56.652 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:26:56.652 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:26:56.652 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:27:05.431 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:05.430+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:27:05.431 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:05.430+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:27:05.432 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:05.430+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:27:05.432 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:05.430+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:27:05.432 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:05.430+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:27:05.444 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:05.442+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:27:05.444 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:05.442+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:27:05.445 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:05.442+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:27:05.445 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:05.442+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:27:05.445 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:05.442+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:27:05.445 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:05.442+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:27:06.663 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:27:06.667 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:27:06.670 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:27:06.672 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:27:07.585 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:27:07.585 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:27:07.586 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506027 2023-10-17T01:27:07.586 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:27:07.586 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:27:07.587 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:27:07.587 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:27:17.598 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:27:17.598 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:27:17.612 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:27:17.612 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:27:18.658 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:27:18.659 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:27:18.659 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506038 2023-10-17T01:27:18.659 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:27:18.660 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:27:18.660 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:27:18.660 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:27:20.630 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:20.626+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:27:20.631 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:20.630+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:27:20.646 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:20.634+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:27:20.647 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:20.638+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:27:20.647 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:20.642+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:27:28.681 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:27:28.681 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:27:28.683 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:27:28.690 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:27:28.962 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:28.962+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:27:28.990 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:28.982+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:27:28.990 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:28.982+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:27:28.991 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:28.982+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:27:28.991 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:28.982+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:27:28.991 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:28.982+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:27:28.991 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:28.982+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:27:28.991 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:28.982+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:27:28.992 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:28.982+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:27:28.992 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:28.982+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:27:28.992 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:28.982+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:27:28.992 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:28.982+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:27:29.607 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:27:29.609 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:27:29.610 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506049 2023-10-17T01:27:29.610 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:27:29.610 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:27:29.611 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:27:29.611 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:27:39.616 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:27:39.618 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:27:39.622 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:27:39.625 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:27:40.469 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:27:40.474 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:27:40.475 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506060 2023-10-17T01:27:40.476 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:27:40.476 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:27:40.476 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:27:40.477 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:27:43.222 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:43.217+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:27:43.222 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:43.217+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:27:43.230 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:43.225+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:27:43.231 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:43.229+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:27:50.479 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:27:50.480 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:27:50.480 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:27:50.482 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:27:51.462 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:27:51.463 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:27:51.463 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506071 2023-10-17T01:27:51.464 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:27:51.464 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:27:51.464 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:27:51.464 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:27:53.844 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:53.841+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:27:53.844 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:53.841+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:27:53.851 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:53.849+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:27:53.851 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:53.849+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:27:53.851 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:53.849+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:27:53.851 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:53.849+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:27:53.851 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:53.849+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:27:53.854 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:53.853+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:27:53.855 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:53.853+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:27:53.855 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:53.853+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:27:53.855 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:53.853+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:27:53.855 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:53.853+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:27:53.855 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:53.853+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:27:53.856 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:53.853+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:27:58.995 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:58.993+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:27:58.995 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:58.993+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:27:58.998 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:58.997+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:27:58.999 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:58.997+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:27:58.999 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:58.997+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:27:58.999 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:58.997+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:27:59.000 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:58.997+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:27:59.005 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:27:59.001+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:28:01.467 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:28:01.469 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:28:01.487 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:28:01.495 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:28:02.377 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:28:02.378 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:28:02.378 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506082 2023-10-17T01:28:02.379 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:28:02.379 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:28:02.379 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:28:02.379 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:28:12.178 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:12.177+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:28:12.181 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:12.177+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:28:12.181 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:12.177+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:28:12.181 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:12.177+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:28:12.181 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:12.177+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:28:12.186 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:12.185+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:28:12.186 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:12.185+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:28:12.187 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:12.185+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:28:12.187 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:12.185+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:28:12.380 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:28:12.384 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:28:12.396 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:28:12.399 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:28:13.286 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:28:13.287 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:28:13.287 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506093 2023-10-17T01:28:13.287 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:28:13.287 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:28:13.288 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:28:13.288 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:28:18.760 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:18.757+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:28:18.760 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:18.757+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:28:18.760 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:18.757+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbda750 2023-10-17T01:28:18.761 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:18.757+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:28:18.761 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:18.757+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:28:18.761 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:18.757+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:28:18.761 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:18.757+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:28:18.761 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:18.757+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:28:18.762 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:18.757+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:28:18.762 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:18.757+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:28:18.762 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:18.757+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:28:18.762 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:18.757+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:28:18.762 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:18.757+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:28:18.762 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:18.757+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:28:18.763 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:18.757+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:28:23.289 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:28:23.290 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:28:23.290 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:28:23.303 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:28:23.670 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:23.665+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:28:23.671 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:23.665+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:28:23.676 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:23.669+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:28:23.676 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:23.669+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:28:23.676 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:23.669+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:28:23.676 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:23.669+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:28:23.691 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:23.689+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:28:23.692 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:23.689+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:28:23.692 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:23.689+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:28:23.692 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:23.689+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:28:23.692 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:23.689+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:28:23.692 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:23.689+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:28:24.240 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:28:24.240 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:28:24.241 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506104 2023-10-17T01:28:24.241 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:28:24.242 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:28:24.242 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:28:24.242 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:28:34.248 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:28:34.256 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:28:34.261 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:28:34.261 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:28:35.060 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:28:35.060 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:28:35.061 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506115 2023-10-17T01:28:35.061 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:28:35.061 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:28:35.061 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:28:35.062 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:28:45.063 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:28:45.063 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:28:45.069 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:28:45.075 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:28:45.943 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:28:45.944 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:28:45.944 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506125 2023-10-17T01:28:45.944 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:28:45.944 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:28:45.945 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:28:45.945 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:28:55.946 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:28:55.947 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:28:55.950 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:28:55.952 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:28:57.037 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:28:57.038 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:28:57.038 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506137 2023-10-17T01:28:57.039 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:28:57.039 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:28:57.039 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:28:57.039 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:28:59.569 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:59.564+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:28:59.585 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:28:59.580+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:29:07.046 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:29:07.047 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:29:07.055 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:29:07.061 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:29:07.961 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:29:07.961 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:29:07.962 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506147 2023-10-17T01:29:07.962 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:29:07.963 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:29:07.963 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:29:07.963 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:29:17.965 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:29:17.965 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:29:17.970 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:29:17.978 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:29:18.976 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:29:18.976 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:29:18.977 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506158 2023-10-17T01:29:18.978 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:29:18.978 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:29:18.978 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:29:18.978 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:29:28.979 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:29:28.980 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:29:28.981 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:29:28.983 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:29:29.841 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:29:29.841 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:29:29.842 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506169 2023-10-17T01:29:29.842 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:29:29.843 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:29:29.843 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:29:29.843 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:29:39.851 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:29:39.851 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:29:39.861 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:29:39.862 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:29:40.699 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:29:40.699 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:29:40.699 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506180 2023-10-17T01:29:40.700 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:29:40.700 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:29:40.700 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:29:40.700 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:29:46.192 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:29:46.187+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:29:46.213 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:29:46.199+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:29:46.214 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:29:46.199+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:29:46.214 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:29:46.199+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:29:46.214 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:29:46.199+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:29:46.215 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:29:46.199+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:29:50.715 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:29:50.716 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:29:50.716 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:29:50.716 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:29:51.736 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:29:51.736 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:29:51.737 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506191 2023-10-17T01:29:51.738 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:29:51.738 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:29:51.738 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:29:51.738 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:30:01.744 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:30:01.745 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:30:01.751 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:30:01.769 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:30:02.802 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:30:02.802 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:30:02.816 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506202 2023-10-17T01:30:02.816 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:30:02.816 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:30:02.817 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:30:02.817 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:30:04.516 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:30:04.511+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:30:04.516 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:30:04.511+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:30:04.517 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:30:04.511+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbda750 2023-10-17T01:30:04.518 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:30:04.511+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:30:04.518 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:30:04.511+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:30:04.519 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:30:04.511+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:30:04.519 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:30:04.511+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:30:04.519 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:30:04.511+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:30:04.519 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:30:04.511+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:30:04.519 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:30:04.511+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:30:04.520 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:30:04.511+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:30:04.520 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:30:04.511+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:30:04.520 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:30:04.511+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:30:04.520 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:30:04.511+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:30:04.520 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:30:04.511+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:30:12.818 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:30:12.819 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:30:12.827 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:30:12.829 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:30:13.725 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:30:13.726 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:30:13.731 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506213 2023-10-17T01:30:13.732 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:30:13.732 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:30:13.732 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:30:13.732 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:30:23.733 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:30:23.738 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:30:23.743 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:30:23.746 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:30:24.655 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:30:24.658 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:30:24.659 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506224 2023-10-17T01:30:24.660 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:30:24.660 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:30:24.660 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:30:24.660 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:30:34.664 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:30:34.664 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:30:34.665 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:30:34.689 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:30:35.582 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:30:35.582 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:30:35.583 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506235 2023-10-17T01:30:35.583 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:30:35.583 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:30:35.583 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:30:35.584 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:30:45.587 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:30:45.591 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:30:45.595 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:30:45.595 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:30:46.572 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:30:46.573 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:30:46.573 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506246 2023-10-17T01:30:46.574 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:30:46.574 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:30:46.574 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:30:46.574 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:30:56.575 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:30:56.576 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:30:56.576 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:30:56.611 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:30:57.424 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:30:57.425 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:30:57.426 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506257 2023-10-17T01:30:57.426 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:30:57.429 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:30:57.430 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:30:57.430 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:31:07.438 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:31:07.452 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:31:07.466 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:31:07.471 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:31:08.468 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:31:08.469 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:31:08.469 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506268 2023-10-17T01:31:08.470 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:31:08.470 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:31:08.470 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:31:08.470 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:31:18.471 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:31:18.471 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:31:18.473 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:31:18.476 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:31:19.450 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:31:19.446+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:31:19.451 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:31:19.446+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:31:19.451 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:31:19.446+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:31:19.458 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:31:19.454+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:31:19.461 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:31:19.458+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:31:19.461 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:31:19.458+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:31:19.461 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:31:19.458+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:31:19.531 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:31:19.532 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:31:19.532 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506279 2023-10-17T01:31:19.532 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:31:19.533 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:31:19.533 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:31:19.533 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:31:29.536 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:31:29.537 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:31:29.552 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:31:29.568 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:31:30.528 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:31:30.529 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:31:30.530 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506290 2023-10-17T01:31:30.530 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:31:30.530 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:31:30.530 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:31:30.531 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:31:40.534 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:31:40.535 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:31:40.535 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:31:40.541 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:31:41.563 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:31:41.563 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:31:41.563 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506301 2023-10-17T01:31:41.564 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:31:41.564 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:31:41.564 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:31:41.564 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:31:48.471 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:31:48.469+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:31:48.486 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:31:48.485+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:31:48.486 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:31:48.485+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:31:50.431 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:31:50.429+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:31:50.457 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:31:50.453+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:31:50.457 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:31:50.453+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbda750 2023-10-17T01:31:50.457 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:31:50.453+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:31:50.457 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:31:50.453+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:31:50.457 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:31:50.453+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:31:50.458 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:31:50.453+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:31:50.458 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:31:50.453+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:31:50.458 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:31:50.453+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:31:50.458 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:31:50.453+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:31:50.458 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:31:50.453+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:31:50.459 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:31:50.453+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:31:50.459 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:31:50.453+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:31:50.459 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:31:50.453+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:31:50.459 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:31:50.453+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:31:51.565 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:31:51.565 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:31:51.587 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:31:51.588 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:31:52.642 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:31:52.646 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:31:52.647 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506312 2023-10-17T01:31:52.647 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:31:52.647 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:31:52.648 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:31:52.648 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:32:02.657 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:32:02.663 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:32:02.666 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:32:02.670 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:32:03.621 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:32:03.621 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:32:03.623 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506323 2023-10-17T01:32:03.623 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:32:03.623 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:32:03.623 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:32:03.623 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:32:08.901 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:32:08.897+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:32:08.901 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:32:08.897+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:32:08.901 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:32:08.897+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:32:08.910 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:32:08.905+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:32:08.910 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:32:08.905+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:32:08.910 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:32:08.905+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:32:08.919 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:32:08.913+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:32:08.938 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:32:08.937+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:32:08.939 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:32:08.937+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:32:08.939 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:32:08.937+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:32:13.627 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:32:13.629 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:32:13.630 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:32:13.637 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:32:14.430 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:32:14.430 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:32:14.431 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506334 2023-10-17T01:32:14.431 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:32:14.432 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:32:14.432 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:32:14.432 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:32:24.432 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:32:24.433 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:32:24.437 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:32:24.442 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:32:25.285 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:32:25.285 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:32:25.286 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506345 2023-10-17T01:32:25.286 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:32:25.286 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:32:25.287 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:32:25.287 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:32:35.290 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:32:35.291 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:32:35.294 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:32:35.297 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:32:36.175 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:32:36.176 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:32:36.176 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506356 2023-10-17T01:32:36.177 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:32:36.177 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:32:36.177 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:32:36.177 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:32:46.179 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:32:46.179 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:32:46.180 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:32:46.182 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:32:47.102 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:32:47.102 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:32:47.103 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506367 2023-10-17T01:32:47.103 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:32:47.103 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:32:47.103 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:32:47.103 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:32:57.109 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:32:57.111 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:32:57.111 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:32:57.119 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:32:57.978 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:32:57.978 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:32:57.979 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506377 2023-10-17T01:32:57.979 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:32:57.979 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:32:57.979 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:32:57.980 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:33:07.985 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:33:07.991 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:33:07.991 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:33:07.999 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:33:08.915 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:33:08.916 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:33:08.916 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506388 2023-10-17T01:33:08.917 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:33:08.917 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:33:08.917 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:33:08.917 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:33:18.918 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:33:18.919 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:33:18.923 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:33:18.925 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:33:19.951 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:33:19.951 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:33:19.952 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506399 2023-10-17T01:33:19.952 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:33:19.952 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:33:19.952 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:33:19.952 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:33:29.953 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:33:29.954 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:33:29.955 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:33:29.957 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:33:30.807 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:33:30.811 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:33:30.812 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506410 2023-10-17T01:33:30.813 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:33:30.813 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:33:30.813 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:33:30.813 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:33:40.817 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:33:40.818 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:33:40.818 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:33:40.821 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:33:41.819 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:33:41.819 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:33:41.819 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506421 2023-10-17T01:33:41.820 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:33:41.820 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:33:41.820 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:33:41.820 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:33:51.822 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:33:51.822 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:33:51.823 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:33:51.854 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:33:52.719 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:33:52.720 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:33:52.720 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506432 2023-10-17T01:33:52.721 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:33:52.721 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:33:52.721 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:33:52.721 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:33:57.977 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:33:57.970+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:33:57.979 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:33:57.974+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:33:57.992 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:33:57.986+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:34:02.728 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:34:02.728 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:34:02.738 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:34:02.739 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:34:03.635 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:34:03.636 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:34:03.636 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506443 2023-10-17T01:34:03.637 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:34:03.637 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:34:03.637 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:34:03.637 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:34:13.639 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:34:13.639 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:34:13.647 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:34:13.664 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:34:14.493 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:34:14.493 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:34:14.494 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506454 2023-10-17T01:34:14.495 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:34:14.495 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:34:14.495 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:34:14.495 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:34:24.509 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:34:24.524 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:34:24.530 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:34:24.536 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:34:25.336 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:34:25.345 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:34:25.346 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506465 2023-10-17T01:34:25.346 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:34:25.346 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:34:25.347 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:34:25.347 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:34:35.359 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:34:35.359 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:34:35.359 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:34:35.367 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:34:36.292 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:34:36.292 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:34:36.293 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506476 2023-10-17T01:34:36.293 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:34:36.293 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:34:36.293 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:34:36.293 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:34:46.295 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:34:46.295 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:34:46.299 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:34:46.302 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:34:47.199 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:34:47.200 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:34:47.201 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506487 2023-10-17T01:34:47.201 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:34:47.201 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:34:47.201 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:34:47.202 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:34:57.203 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:34:57.203 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:34:57.215 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:34:57.226 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:34:58.060 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:34:58.060 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:34:58.061 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506498 2023-10-17T01:34:58.061 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:34:58.061 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:34:58.062 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:34:58.062 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:35:08.067 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:35:08.067 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:35:08.068 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:35:08.089 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:35:09.004 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:35:09.018 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:35:09.027 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506509 2023-10-17T01:35:09.027 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:35:09.028 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:35:09.028 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:35:09.028 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:35:18.608 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:35:18.605+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:35:18.609 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:35:18.605+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:35:18.609 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:35:18.605+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:35:19.040 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:35:19.041 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:35:19.049 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:35:19.053 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:35:19.967 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:35:19.975 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:35:19.976 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506519 2023-10-17T01:35:19.977 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:35:19.977 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:35:19.977 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:35:19.978 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:35:29.970 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:35:29.983 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:35:29.988 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:35:29.996 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:35:30.872 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:35:30.873 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:35:30.873 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506530 2023-10-17T01:35:30.874 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:35:30.874 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:35:30.874 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:35:30.874 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:35:34.709 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:35:34.704+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:35:34.710 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:35:34.704+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:35:34.712 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:35:34.708+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:35:34.713 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:35:34.712+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:35:40.875 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:35:40.877 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:35:40.885 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:35:40.889 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:35:41.828 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:35:41.828 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:35:41.829 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506541 2023-10-17T01:35:41.829 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:35:41.829 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:35:41.830 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:35:41.830 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:35:47.354 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:35:47.352+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:35:47.355 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:35:47.352+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:35:47.356 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:35:47.352+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:35:47.356 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:35:47.352+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:35:47.360 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:35:47.356+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:35:47.364 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:35:47.360+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:35:47.367 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:35:47.364+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:35:51.834 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:35:51.834 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:35:51.835 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:35:51.857 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:35:52.704 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:35:52.709 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:35:52.710 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506552 2023-10-17T01:35:52.710 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:35:52.710 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:35:52.711 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:35:52.711 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:36:02.712 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:36:02.714 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:36:02.726 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:36:02.734 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:36:03.656 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:36:03.657 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:36:03.657 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506563 2023-10-17T01:36:03.657 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:36:03.658 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:36:03.658 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:36:03.658 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:36:13.669 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:36:13.674 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:36:13.687 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:36:13.692 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:36:14.610 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:36:14.610 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:36:14.611 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506574 2023-10-17T01:36:14.611 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:36:14.611 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:36:14.611 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:36:14.612 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:36:24.613 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:36:24.613 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:36:24.614 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:36:24.616 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:36:25.591 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:36:25.591 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:36:25.592 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506585 2023-10-17T01:36:25.592 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:36:25.592 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:36:25.593 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:36:25.593 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:36:35.594 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:36:35.597 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:36:35.598 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:36:35.621 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:36:36.591 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:36:36.591 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:36:36.592 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506596 2023-10-17T01:36:36.592 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:36:36.592 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:36:36.593 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:36:36.593 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:36:42.882 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:36:42.879+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:36:42.882 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:36:42.879+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:36:42.883 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:36:42.879+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:36:42.885 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:36:42.879+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:36:46.593 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:36:46.594 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:36:46.597 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:36:46.617 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:36:47.474 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:36:47.474 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:36:47.475 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506607 2023-10-17T01:36:47.475 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:36:47.475 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:36:47.475 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:36:47.475 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:36:57.475 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:36:57.477 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:36:57.485 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:36:57.488 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:36:58.351 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:36:58.352 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:36:58.353 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506618 2023-10-17T01:36:58.353 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:36:58.354 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:36:58.354 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:36:58.354 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:37:08.370 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:37:08.370 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:37:08.370 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:37:08.373 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:37:09.365 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:37:09.366 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:37:09.366 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506629 2023-10-17T01:37:09.367 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:37:09.367 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:37:09.367 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:37:09.367 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:37:19.353 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:37:19.346+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:37:19.356 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:37:19.354+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:37:19.362 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:37:19.362+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:37:19.371 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:37:19.371 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:37:19.376 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:37:19.383 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:37:20.129 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:37:20.129 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:37:20.130 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506640 2023-10-17T01:37:20.130 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:37:20.130 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:37:20.131 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:37:20.131 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:37:30.132 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:37:30.133 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:37:30.135 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:37:30.138 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:37:30.935 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:37:30.936 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:37:30.936 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506650 2023-10-17T01:37:30.937 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:37:30.937 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:37:30.937 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:37:30.937 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:37:40.952 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:37:40.952 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:37:40.955 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:37:40.965 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:37:41.983 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:37:41.983 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:37:41.984 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506661 2023-10-17T01:37:41.984 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:37:41.984 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:37:41.984 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:37:41.985 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:37:51.989 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:37:51.991 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:37:51.993 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:37:52.003 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:37:52.894 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:37:52.894 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:37:52.895 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506672 2023-10-17T01:37:52.895 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:37:52.895 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:37:52.895 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:37:52.896 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:38:02.901 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:38:02.902 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:38:02.902 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:38:02.903 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:38:03.877 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:38:03.877 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:38:03.878 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506683 2023-10-17T01:38:03.878 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:38:03.878 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:38:03.878 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:38:03.878 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:38:13.900 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:38:13.901 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:38:13.902 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:38:13.903 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:38:14.764 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:38:14.764 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:38:14.765 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506694 2023-10-17T01:38:14.765 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:38:14.765 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:38:14.765 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:38:14.765 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:38:24.766 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:38:24.770 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:38:24.773 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:38:24.796 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:38:25.721 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:38:25.722 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:38:25.723 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506705 2023-10-17T01:38:25.723 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:38:25.723 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:38:25.723 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:38:25.723 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:38:35.724 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:38:35.724 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:38:35.729 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:38:35.735 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:38:36.753 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:38:36.753 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:38:36.754 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506716 2023-10-17T01:38:36.754 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:38:36.754 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:38:36.754 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:38:36.755 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:38:46.762 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:38:46.763 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:38:46.763 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:38:46.796 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:38:47.669 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:38:47.670 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:38:47.670 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506727 2023-10-17T01:38:47.670 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:38:47.671 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:38:47.671 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:38:47.671 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:38:57.680 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:38:57.686 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:38:57.686 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:38:57.704 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:38:58.637 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:38:58.639 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:38:58.641 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506738 2023-10-17T01:38:58.641 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:38:58.641 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:38:58.641 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:38:58.642 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:39:08.643 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:39:08.643 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:39:08.643 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:39:08.649 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:39:09.569 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:39:09.570 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:39:09.570 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506749 2023-10-17T01:39:09.570 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:39:09.571 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:39:09.571 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:39:09.571 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:39:19.573 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:39:19.573 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:39:19.574 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:39:19.583 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:39:20.468 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:39:20.468 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:39:20.468 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506760 2023-10-17T01:39:20.469 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:39:20.469 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:39:20.469 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:39:20.469 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:39:30.471 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:39:30.472 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:39:30.472 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:39:30.482 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:39:31.433 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:39:31.433 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:39:31.434 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506771 2023-10-17T01:39:31.434 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:39:31.434 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:39:31.435 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:39:31.435 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:39:41.436 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:39:41.436 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:39:41.437 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:39:41.444 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:39:42.442 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:39:42.442 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:39:42.442 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506782 2023-10-17T01:39:42.443 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:39:42.443 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:39:42.443 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:39:42.443 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:39:52.445 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:39:52.738 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:39:52.738 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:39:52.738 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:39:53.364 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:39:53.364 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:39:53.365 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506793 2023-10-17T01:39:53.365 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:39:53.366 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:39:53.366 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:39:53.366 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:40:03.369 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:40:03.369 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:40:03.374 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:40:03.374 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:40:04.317 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:40:04.317 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:40:04.317 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506804 2023-10-17T01:40:04.318 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:40:04.318 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:40:04.318 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:40:04.318 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:40:08.355 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:40:08.346+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:40:08.362 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:40:08.350+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:40:08.363 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:40:08.350+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:40:08.363 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:40:08.354+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:40:08.363 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:40:08.354+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:40:08.363 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:40:08.354+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:40:08.379 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:40:08.366+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:40:08.379 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:40:08.366+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:40:08.380 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:40:08.366+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:40:08.380 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:40:08.366+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:40:14.320 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:40:14.321 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:40:14.321 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:40:14.337 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:40:15.155 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:40:15.155 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:40:15.156 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506815 2023-10-17T01:40:15.156 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:40:15.156 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:40:15.156 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:40:15.157 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:40:25.157 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:40:25.158 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:40:25.171 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:40:25.172 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:40:26.078 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:40:26.079 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:40:26.080 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506826 2023-10-17T01:40:26.080 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:40:26.080 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:40:26.080 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:40:26.080 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:40:36.086 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:40:36.087 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:40:36.089 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:40:36.110 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:40:36.969 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:40:36.978 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:40:36.979 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506836 2023-10-17T01:40:36.980 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:40:36.980 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:40:36.980 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:40:36.980 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:40:46.992 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:40:46.999 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:40:47.005 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:40:47.010 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:40:48.293 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:40:48.296 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:40:48.297 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506848 2023-10-17T01:40:48.297 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:40:48.297 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:40:48.297 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:40:48.297 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:40:58.300 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:40:58.312 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:40:58.313 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:40:58.316 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:40:59.229 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:40:59.230 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:40:59.230 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506859 2023-10-17T01:40:59.230 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:40:59.231 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:40:59.231 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:40:59.231 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:41:07.682 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:41:07.677+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:41:07.689 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:41:07.685+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:41:07.691 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:41:07.689+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:41:09.239 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:41:09.240 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:41:09.240 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:41:09.246 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:41:10.157 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:41:10.157 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:41:10.158 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506870 2023-10-17T01:41:10.158 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:41:10.158 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:41:10.159 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:41:10.159 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:41:20.160 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:41:20.167 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:41:20.170 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:41:20.182 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:41:21.086 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:41:21.087 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:41:21.087 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506881 2023-10-17T01:41:21.087 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:41:21.087 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:41:21.088 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:41:21.088 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:41:28.719 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:41:28.717+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:41:28.719 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:41:28.717+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:41:28.734 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:41:28.733+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:41:28.734 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:41:28.733+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:41:31.092 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:41:31.095 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:41:31.097 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:41:31.118 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:41:32.035 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:41:32.040 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:41:32.041 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506892 2023-10-17T01:41:32.042 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:41:32.042 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:41:32.042 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:41:32.042 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:41:38.192 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:41:38.189+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:41:38.193 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:41:38.189+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:41:38.193 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:41:38.189+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbda750 2023-10-17T01:41:38.193 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:41:38.189+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:41:38.193 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:41:38.189+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:41:38.193 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:41:38.189+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:41:38.193 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:41:38.189+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:41:38.194 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:41:38.189+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:41:38.194 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:41:38.189+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:41:38.194 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:41:38.189+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:41:38.194 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:41:38.189+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:41:38.194 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:41:38.189+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:41:38.195 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:41:38.189+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:41:38.195 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:41:38.189+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:41:38.195 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:41:38.189+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:41:38.365 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:41:38.361+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:41:38.366 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:41:38.361+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:41:38.382 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:41:38.373+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:41:38.382 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:41:38.373+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:41:42.047 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:41:42.047 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:41:42.050 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:41:42.053 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:41:42.854 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:41:42.855 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:41:42.855 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506902 2023-10-17T01:41:42.856 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:41:42.856 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:41:42.856 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:41:42.856 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:41:52.857 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:41:52.862 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:41:52.862 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:41:52.865 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:41:53.772 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:41:53.772 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:41:53.773 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506913 2023-10-17T01:41:53.773 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:41:53.773 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:41:53.773 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:41:53.773 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:42:03.775 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:42:03.779 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:42:03.792 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:42:03.800 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:42:04.824 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:42:04.824 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:42:04.825 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506924 2023-10-17T01:42:04.825 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:42:04.825 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:42:04.825 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:42:04.825 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:42:14.830 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:42:14.833 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:42:14.834 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:42:14.834 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:42:15.177 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:42:15.172+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:42:15.191 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:42:15.184+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:42:15.198 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:42:15.196+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:42:15.199 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:42:15.196+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:42:15.199 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:42:15.196+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:42:15.199 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:42:15.196+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:42:15.199 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:42:15.196+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:42:15.200 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:42:15.196+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:42:15.200 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:42:15.196+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:42:15.200 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:42:15.196+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:42:15.200 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:42:15.196+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:42:15.200 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:42:15.196+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:42:15.201 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:42:15.196+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:42:15.201 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:42:15.196+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:42:15.756 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:42:15.756 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:42:15.757 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506935 2023-10-17T01:42:15.757 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:42:15.757 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:42:15.757 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:42:15.758 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:42:25.765 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:42:25.768 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:42:25.777 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:42:25.782 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:42:26.665 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:42:26.665 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:42:26.666 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506946 2023-10-17T01:42:26.666 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:42:26.667 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:42:26.667 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:42:26.667 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:42:36.677 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:42:36.680 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:42:36.681 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:42:36.683 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:42:37.612 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:42:37.612 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:42:37.613 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506957 2023-10-17T01:42:37.613 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:42:37.614 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:42:37.614 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:42:37.614 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:42:47.313 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:42:47.307+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:42:47.313 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:42:47.307+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:42:47.314 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:42:47.307+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:42:47.314 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:42:47.307+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:42:47.319 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:42:47.315+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:42:47.319 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:42:47.315+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:42:47.319 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:42:47.315+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:42:47.615 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:42:47.615 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:42:47.617 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:42:47.619 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:42:48.532 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:42:48.532 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:42:48.533 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506968 2023-10-17T01:42:48.533 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:42:48.533 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:42:48.534 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:42:48.534 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:42:58.578 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:42:58.578 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:42:58.579 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:42:58.598 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:42:59.525 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:42:59.526 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:42:59.526 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506979 2023-10-17T01:42:59.526 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:42:59.527 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:42:59.527 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:42:59.527 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:43:04.425 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:43:04.419+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:43:04.427 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:43:04.419+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:43:09.531 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:43:09.533 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:43:09.538 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:43:09.539 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:43:10.597 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:43:10.597 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:43:10.616 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697506990 2023-10-17T01:43:10.616 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:43:10.617 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:43:10.617 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:43:10.617 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:43:15.321 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:43:15.307+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:43:15.322 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:43:15.319+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:43:15.328 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:43:15.323+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:43:15.338 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:43:15.335+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:43:15.338 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:43:15.335+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:43:15.338 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:43:15.335+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:43:15.338 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:43:15.335+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:43:15.339 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:43:15.335+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:43:15.339 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:43:15.335+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:43:15.339 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:43:15.335+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:43:15.339 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:43:15.335+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:43:15.339 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:43:15.335+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:43:15.340 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:43:15.335+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:43:15.340 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:43:15.335+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:43:20.622 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:43:20.622 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:43:20.627 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:43:20.631 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:43:21.571 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:43:21.571 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:43:21.573 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697507001 2023-10-17T01:43:21.573 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:43:21.573 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:43:21.573 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:43:21.574 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:43:31.575 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:43:31.576 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:43:31.585 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:43:31.598 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:43:32.454 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:43:32.454 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:43:32.454 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697507012 2023-10-17T01:43:32.455 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:43:32.455 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:43:32.455 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:43:32.455 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:43:38.065 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:43:38.058+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:43:38.065 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:43:38.058+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:43:38.066 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:43:38.058+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:43:38.068 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:43:38.062+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:43:38.068 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:43:38.062+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:43:38.072 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:43:38.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:43:38.072 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:43:38.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:43:38.072 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:43:38.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:43:38.073 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:43:38.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:43:42.456 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:43:42.457 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:43:42.457 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:43:42.462 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:43:43.497 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:43:43.497 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:43:43.498 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697507023 2023-10-17T01:43:43.498 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:43:43.498 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ -n 172.21.15.169:0/1398705567 ]] 2023-10-17T01:43:43.498 INFO:tasks.workunit.client.0.smithi169.stderr:+ [[ 172.21.15.169:0/1398705567 != \1\7\2\.\2\1\.\1\5\.\1\6\9\:\0\/\1\3\9\8\7\0\5\5\6\7 ]] 2023-10-17T01:43:43.499 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:43:53.502 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-17T01:43:53.503 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq '.active_clients[]' 2023-10-17T01:43:53.512 INFO:tasks.workunit.client.0.smithi169.stderr:++ ceph mgr dump 2023-10-17T01:43:53.519 INFO:tasks.workunit.client.0.smithi169.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-17T01:43:54.426 INFO:tasks.workunit.client.0.smithi169.stderr:+ CLIENT_ADDR=172.21.15.169:0/1398705567 2023-10-17T01:43:54.427 INFO:tasks.workunit.client.0.smithi169.stderr:++ date +%s 2023-10-17T01:43:54.427 INFO:tasks.workunit.client.0.smithi169.stderr:+ CURRENT_TIME=1697507034 2023-10-17T01:43:54.428 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-17T01:43:54.428 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i = 1 )) 2023-10-17T01:43:54.428 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 24 )) 2023-10-17T01:43:54.428 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-17T01:43:54.686 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:43:54.686+0000 7f34f98be640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-17T01:43:54.686 INFO:tasks.workunit.client.0.smithi169.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-17T01:43:54.692 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:44:00.104 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:44:00.098+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:44:04.693 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T01:44:04.693 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 24 )) 2023-10-17T01:44:04.694 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-17T01:44:04.784 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:44:04.782+0000 7f34f98be640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-17T01:44:04.784 INFO:tasks.workunit.client.0.smithi169.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-17T01:44:04.790 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:44:13.400 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:44:13.398+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:44:13.400 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:44:13.398+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:44:13.400 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:44:13.398+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:44:14.792 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T01:44:14.792 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 24 )) 2023-10-17T01:44:14.792 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-17T01:44:14.907 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:44:14.906+0000 7f34f98be640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-17T01:44:14.907 INFO:tasks.workunit.client.0.smithi169.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-17T01:44:14.914 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:44:24.915 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T01:44:24.915 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 24 )) 2023-10-17T01:44:24.916 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-17T01:44:25.008 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:44:25.005+0000 7f34f98be640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-17T01:44:25.008 INFO:tasks.workunit.client.0.smithi169.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-17T01:44:25.026 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:44:35.031 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T01:44:35.031 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 24 )) 2023-10-17T01:44:35.031 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-17T01:44:35.105 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:44:35.101+0000 7f34f98be640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-17T01:44:35.105 INFO:tasks.workunit.client.0.smithi169.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-17T01:44:35.110 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:44:45.112 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T01:44:45.112 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 24 )) 2023-10-17T01:44:45.113 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-17T01:44:45.201 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:44:45.201+0000 7f34f98be640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-17T01:44:45.202 INFO:tasks.workunit.client.0.smithi169.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-17T01:44:45.210 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:44:55.211 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T01:44:55.211 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 24 )) 2023-10-17T01:44:55.211 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-17T01:44:55.310 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:44:55.309+0000 7f34f98be640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-17T01:44:55.311 INFO:tasks.workunit.client.0.smithi169.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-17T01:44:55.320 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:45:05.321 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T01:45:05.322 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 24 )) 2023-10-17T01:45:05.322 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-17T01:45:05.391 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:45:05.389+0000 7f34f98be640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-17T01:45:05.391 INFO:tasks.workunit.client.0.smithi169.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-17T01:45:05.395 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:45:15.399 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T01:45:15.425 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 24 )) 2023-10-17T01:45:15.425 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-17T01:45:15.505 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:45:15.500+0000 7f34f98be640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-17T01:45:15.505 INFO:tasks.workunit.client.0.smithi169.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-17T01:45:15.512 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:45:25.514 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T01:45:25.515 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 24 )) 2023-10-17T01:45:25.515 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-17T01:45:25.655 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:45:25.652+0000 7f34f98be640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-17T01:45:25.664 INFO:tasks.workunit.client.0.smithi169.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-17T01:45:25.676 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:45:35.678 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T01:45:35.678 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 24 )) 2023-10-17T01:45:35.678 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-17T01:45:35.825 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:45:35.824+0000 7f34f98be640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-17T01:45:35.826 INFO:tasks.workunit.client.0.smithi169.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-17T01:45:35.836 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:45:45.838 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T01:45:45.838 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 24 )) 2023-10-17T01:45:45.839 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-17T01:45:45.910 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:45:45.908+0000 7f34f98be640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-17T01:45:45.911 INFO:tasks.workunit.client.0.smithi169.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-17T01:45:45.918 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:45:55.927 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T01:45:55.928 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 24 )) 2023-10-17T01:45:55.928 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-17T01:45:55.997 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:45:55.992+0000 7f34f98be640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-17T01:45:55.998 INFO:tasks.workunit.client.0.smithi169.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-17T01:45:56.001 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:46:06.002 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T01:46:06.003 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 24 )) 2023-10-17T01:46:06.003 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-17T01:46:06.095 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:46:06.091+0000 7f34f98be640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-17T01:46:06.096 INFO:tasks.workunit.client.0.smithi169.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-17T01:46:06.109 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:46:16.111 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T01:46:16.111 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 24 )) 2023-10-17T01:46:16.111 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-17T01:46:16.235 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:46:16.231+0000 7f34f98be640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-17T01:46:16.236 INFO:tasks.workunit.client.0.smithi169.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-17T01:46:16.248 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:46:17.328 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:46:17.319+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:46:17.329 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:46:17.323+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:46:17.357 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:46:17.347+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:46:17.358 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:46:17.347+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:46:17.358 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:46:17.347+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:46:17.358 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:46:17.347+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:46:17.358 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:46:17.347+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:46:17.359 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:46:17.355+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:46:17.360 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:46:17.355+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:46:17.360 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:46:17.355+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:46:17.367 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:46:17.363+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:46:17.377 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:46:17.367+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:46:17.377 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:46:17.367+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:46:17.378 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:46:17.367+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:46:26.249 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T01:46:26.249 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 24 )) 2023-10-17T01:46:26.250 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-17T01:46:26.320 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:46:26.315+0000 7f34f98be640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-17T01:46:26.331 INFO:tasks.workunit.client.0.smithi169.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-17T01:46:26.341 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:46:36.347 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T01:46:36.348 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 24 )) 2023-10-17T01:46:36.348 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-17T01:46:36.458 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:46:36.451+0000 7f34f98be640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-17T01:46:36.458 INFO:tasks.workunit.client.0.smithi169.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-17T01:46:36.461 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:46:46.463 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T01:46:46.463 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 24 )) 2023-10-17T01:46:46.463 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-17T01:46:46.525 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:46:46.518+0000 7f34f98be640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-17T01:46:46.526 INFO:tasks.workunit.client.0.smithi169.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-17T01:46:46.548 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:46:55.071 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:46:55.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:46:55.072 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:46:55.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:46:55.072 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:46:55.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:46:55.072 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:46:55.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:46:55.072 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:46:55.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:46:55.072 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:46:55.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:46:55.081 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:46:55.074+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:46:55.082 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:46:55.074+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:46:55.095 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:46:55.090+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:46:55.095 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:46:55.090+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:46:55.096 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:46:55.090+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:46:56.550 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T01:46:56.550 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 24 )) 2023-10-17T01:46:56.550 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-17T01:46:56.643 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:46:56.638+0000 7f34f98be640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-17T01:46:56.643 INFO:tasks.workunit.client.0.smithi169.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-17T01:46:56.647 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:47:06.649 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T01:47:06.649 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 24 )) 2023-10-17T01:47:06.649 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-17T01:47:06.800 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:47:06.798+0000 7f34f98be640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-17T01:47:06.800 INFO:tasks.workunit.client.0.smithi169.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-17T01:47:06.816 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:47:16.818 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T01:47:16.818 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 24 )) 2023-10-17T01:47:16.818 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-17T01:47:16.893 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:47:16.890+0000 7f34f98be640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-17T01:47:16.894 INFO:tasks.workunit.client.0.smithi169.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-17T01:47:16.908 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:47:26.910 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T01:47:26.910 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 24 )) 2023-10-17T01:47:26.910 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-17T01:47:26.987 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:47:26.986+0000 7f34f98be640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-17T01:47:26.993 INFO:tasks.workunit.client.0.smithi169.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-17T01:47:27.001 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:47:37.005 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T01:47:37.006 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 24 )) 2023-10-17T01:47:37.006 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-17T01:47:37.133 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:47:37.129+0000 7f34f98be640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-17T01:47:37.134 INFO:tasks.workunit.client.0.smithi169.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-17T01:47:37.152 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:47:45.343 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:47:45.333+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:47:45.353 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:47:45.349+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:47:45.361 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:47:45.357+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbda750 2023-10-17T01:47:45.362 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:47:45.357+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:47:45.362 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:47:45.357+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:47:45.362 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:47:45.357+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:47:45.362 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:47:45.357+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:47:45.362 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:47:45.357+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:47:45.363 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:47:45.357+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:47:45.363 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:47:45.357+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:47:45.363 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:47:45.357+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:47:45.363 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:47:45.357+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:47:45.364 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:47:45.357+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:47:45.364 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:47:45.357+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:47:45.364 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:47:45.357+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:47:47.153 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T01:47:47.153 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 24 )) 2023-10-17T01:47:47.154 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-17T01:47:47.213 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:47:47.209+0000 7f34f98be640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-17T01:47:47.213 INFO:tasks.workunit.client.0.smithi169.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-17T01:47:47.217 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 10 2023-10-17T01:47:54.906 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:47:54.901+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:47:54.907 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:47:54.901+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:47:54.916 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:47:54.909+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:47:57.218 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i++ )) 2023-10-17T01:47:57.218 INFO:tasks.workunit.client.0.smithi169.stderr:+ (( i <= 24 )) 2023-10-17T01:47:57.219 INFO:tasks.workunit.client.0.smithi169.stderr:+ grep 'every 2m' 2023-10-17T01:47:57.232 INFO:tasks.workunit.client.0.smithi169.stderr:+ rbd mirror snapshot schedule ls -p rbd --image image1 2023-10-17T01:47:57.318 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:47:57.317+0000 7f34f98be640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-17T01:47:57.318 INFO:tasks.workunit.client.0.smithi169.stderr:rbd: rbd mirror snapshot schedule list failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-17T01:47:57.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-17T01:47:57.409 INFO:tasks.workunit:Stopping ['rbd/rbd_support_module_recovery.sh'] on client.0... 2023-10-17T01:47:57.409 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-10-17T01:47:58.024 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_c3822f5f81e37dae2c0291e57dc72240858225fd/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_c3822f5f81e37dae2c0291e57dc72240858225fd/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi169 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c3822f5f81e37dae2c0291e57dc72240858225fd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-10-17T01:47:58.090 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=8ca8d2a4f81a409788e300315003a09a Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_c3822f5f81e37dae2c0291e57dc72240858225fd/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_c3822f5f81e37dae2c0291e57dc72240858225fd/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi169 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c3822f5f81e37dae2c0291e57dc72240858225fd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-10-17T01:47:58.093 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-17T01:47:58.137 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-17T01:47:58.147 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-17T01:47:58.147 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:47:58.697 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:47:58.699 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:47:58.730 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2068,"stamp":"2023-10-17T01:47:57.329848+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26690939736,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1747249,"num_read_kb":6795021,"num_write":1719185,"num_write_kb":6876391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81538,"ondisk_log_size":81538,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":17891004,"kb_used_data":13744612,"kb_used_omap":1134,"kb_used_meta":4145233,"kb_avail":265224516,"statfs":{"total":289910292480,"available":271589904384,"internally_reserved":0,"allocated":14074482688,"data_stored":17474468633,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1161646,"internal_metadata":4244719186},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,5,8,16,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16678912,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3875,"num_read_kb":15476,"num_write":4084,"num_write_kb":16336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002102"},"pg_stats":[{"pgid":"2.7","version":"95'11211","reported_seq":16037,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:47:55.264067+0000","last_change":"2023-10-17T00:42:12.798254+0000","last_active":"2023-10-17T01:47:55.264067+0000","last_peered":"2023-10-17T01:47:55.264067+0000","last_clean":"2023-10-17T01:47:55.264067+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:47:55.264067+0000","last_undegraded":"2023-10-17T01:47:55.264067+0000","last_fullsized":"2023-10-17T01:47:55.264067+0000","mapping_epoch":13,"log_start":"50'1200","ondisk_log_start":"50'1200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:46:37.107206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"95'8444","reported_seq":13231,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:47:55.263437+0000","last_change":"2023-10-17T00:42:12.800590+0000","last_active":"2023-10-17T01:47:55.263437+0000","last_peered":"2023-10-17T01:47:55.263437+0000","last_clean":"2023-10-17T01:47:55.263437+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:47:55.263437+0000","last_undegraded":"2023-10-17T01:47:55.263437+0000","last_fullsized":"2023-10-17T01:47:55.263437+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":8444,"log_dups_size":0,"ondisk_log_size":8444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T09:24:36.749724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"95'5661","reported_seq":8146,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:47:55.263145+0000","last_change":"2023-10-17T00:42:12.798876+0000","last_active":"2023-10-17T01:47:55.263145+0000","last_peered":"2023-10-17T01:47:55.263145+0000","last_clean":"2023-10-17T01:47:55.263145+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:47:55.263145+0000","last_undegraded":"2023-10-17T01:47:55.263145+0000","last_fullsized":"2023-10-17T01:47:55.263145+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":5661,"log_dups_size":0,"ondisk_log_size":5661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:08:55.627747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"95'8468","reported_seq":12660,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:47:55.263263+0000","last_change":"2023-10-17T00:42:12.796917+0000","last_active":"2023-10-17T01:47:55.263263+0000","last_peered":"2023-10-17T01:47:55.263263+0000","last_clean":"2023-10-17T01:47:55.263263+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:47:55.263263+0000","last_undegraded":"2023-10-17T01:47:55.263263+0000","last_fullsized":"2023-10-17T01:47:55.263263+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":8468,"log_dups_size":0,"ondisk_log_size":8468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:54:27.448734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"95'436673","reported_seq":871667,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:47:54.867674+0000","last_change":"2023-10-17T00:42:20.880933+0000","last_active":"2023-10-17T01:47:54.867674+0000","last_peered":"2023-10-17T01:47:54.867674+0000","last_clean":"2023-10-17T01:47:54.867674+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:47:54.867674+0000","last_undegraded":"2023-10-17T01:47:54.867674+0000","last_fullsized":"2023-10-17T01:47:54.867674+0000","mapping_epoch":16,"log_start":"93'426608","ondisk_log_start":"93'426608","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:20:26.021756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":6764609536,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":436146,"num_read_kb":1713784,"num_write":435375,"num_write_kb":1741500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"64'32","reported_seq":222,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-17T01:47:09.549761+0000","last_change":"2023-10-17T00:42:12.797141+0000","last_active":"2023-10-17T01:47:09.549761+0000","last_peered":"2023-10-17T01:47:09.549761+0000","last_clean":"2023-10-17T01:47:09.549761+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:47:09.549761+0000","last_undegraded":"2023-10-17T01:47:09.549761+0000","last_fullsized":"2023-10-17T01:47:09.549761+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:01:11.302607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"95'420469","reported_seq":840068,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:47:55.337291+0000","last_change":"2023-10-17T00:42:20.886926+0000","last_active":"2023-10-17T01:47:55.337291+0000","last_peered":"2023-10-17T01:47:55.337291+0000","last_clean":"2023-10-17T01:47:55.337291+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:47:55.337291+0000","last_undegraded":"2023-10-17T01:47:55.337291+0000","last_fullsized":"2023-10-17T01:47:55.337291+0000","mapping_epoch":16,"log_start":"93'410416","ondisk_log_start":"93'410416","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:20:34.646714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6506606592,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":420699,"num_read_kb":1654904,"num_write":419221,"num_write_kb":1676884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"91'61","reported_seq":2433,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:47:50.144161+0000","last_change":"2023-10-17T00:42:12.799004+0000","last_active":"2023-10-17T01:47:50.144161+0000","last_peered":"2023-10-17T01:47:50.144161+0000","last_clean":"2023-10-17T01:47:50.144161+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:47:50.144161+0000","last_undegraded":"2023-10-17T01:47:50.144161+0000","last_fullsized":"2023-10-17T01:47:50.144161+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:21:48.553692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"95'438313","reported_seq":874391,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:47:54.873723+0000","last_change":"2023-10-17T00:42:20.880768+0000","last_active":"2023-10-17T01:47:54.873723+0000","last_peered":"2023-10-17T01:47:54.873723+0000","last_clean":"2023-10-17T01:47:54.873723+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:47:54.873723+0000","last_undegraded":"2023-10-17T01:47:54.873723+0000","last_fullsized":"2023-10-17T01:47:54.873723+0000","mapping_epoch":16,"log_start":"93'428217","ondisk_log_start":"93'428217","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:38:33.949294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":6784278528,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":437235,"num_read_kb":1718856,"num_write":437012,"num_write_kb":1748048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"91'103","reported_seq":3666,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:47:50.144233+0000","last_change":"2023-10-17T00:42:12.799218+0000","last_active":"2023-10-17T01:47:50.144233+0000","last_peered":"2023-10-17T01:47:50.144233+0000","last_clean":"2023-10-17T01:47:50.144233+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:47:50.144233+0000","last_undegraded":"2023-10-17T01:47:50.144233+0000","last_fullsized":"2023-10-17T01:47:50.144233+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:00:25.745996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"95'428220","reported_seq":856297,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:47:55.337113+0000","last_change":"2023-10-17T00:42:20.888387+0000","last_active":"2023-10-17T01:47:55.337113+0000","last_peered":"2023-10-17T01:47:55.337113+0000","last_clean":"2023-10-17T01:47:55.337113+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:47:55.337113+0000","last_undegraded":"2023-10-17T01:47:55.337113+0000","last_fullsized":"2023-10-17T01:47:55.337113+0000","mapping_epoch":16,"log_start":"93'418216","ondisk_log_start":"93'418216","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:38:25.375110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6634852371,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":429207,"num_read_kb":1686824,"num_write":426947,"num_write_kb":1707782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"95'8443","reported_seq":11999,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:47:55.263816+0000","last_change":"2023-10-17T00:42:12.796559+0000","last_active":"2023-10-17T01:47:55.263816+0000","last_peered":"2023-10-17T01:47:55.263816+0000","last_clean":"2023-10-17T01:47:55.263816+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:47:55.263816+0000","last_undegraded":"2023-10-17T01:47:55.263816+0000","last_fullsized":"2023-10-17T01:47:55.263816+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":8443,"log_dups_size":0,"ondisk_log_size":8443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:59:45.094919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":253,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-17T01:46:34.868695+0000","last_change":"2023-10-17T00:42:10.044091+0000","last_active":"2023-10-17T01:46:34.868695+0000","last_peered":"2023-10-17T01:46:34.868695+0000","last_clean":"2023-10-17T01:46:34.868695+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:46:34.868695+0000","last_undegraded":"2023-10-17T01:46:34.868695+0000","last_fullsized":"2023-10-17T01:46:34.868695+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:08.753993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:08.753993+0000","last_clean_scrub_stamp":"2023-10-17T00:42:08.753993+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:18:21.220207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26690347027,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1723287,"num_read_kb":6774368,"num_write":1718555,"num_write_kb":6874214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14074269696,"data_stored":17475129382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40218,"ondisk_log_size":40218,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":41223,"ondisk_log_size":41223,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739161,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8428716,"kb_used_data":6869796,"kb_used_omap":560,"kb_used_meta":1558351,"kb_avail":85943124,"statfs":{"total":96636764160,"available":88005758976,"internally_reserved":0,"allocated":7034671104,"data_stored":8734535516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":574073,"internal_metadata":1595751815},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:46:59 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.30599999999999999,"15min":0.35999999999999999},"min":{"1min":0.183,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38100000000000001,"5min":0.93100000000000005,"15min":4.5030000000000001},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.437,"15min":0.52400000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.183},"max":{"1min":4.1959999999999997,"5min":4.1959999999999997,"15min":5.4240000000000004},"last":0.41399999999999998}]},{"osd":1,"last update":"Tue Oct 17 01:47:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.58299999999999996,"15min":0.497},"min":{"1min":0.223,"5min":0.19400000000000001,"15min":0.191},"max":{"1min":2.7799999999999998,"5min":12.413,"15min":12.413},"last":0.307},{"interface":"front","average":{"1min":0.497,"5min":0.58199999999999996,"15min":0.47899999999999998},"min":{"1min":0.26400000000000001,"5min":0.223,"15min":0.223},"max":{"1min":2.7480000000000002,"5min":5.0529999999999999,"15min":5.282},"last":0.38400000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739159,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4593124,"kb_used_data":3384988,"kb_used_omap":284,"kb_used_meta":1207843,"kb_avail":89778716,"statfs":{"total":96636764160,"available":91933405184,"internally_reserved":0,"allocated":3466227712,"data_stored":4303865723,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":291170,"internal_metadata":1236831902},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:47:45 2023","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":0.64100000000000001,"15min":0.45100000000000001},"min":{"1min":0.17799999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":23.177,"5min":23.177,"15min":23.177},"last":0.32400000000000001},{"interface":"front","average":{"1min":1.6579999999999999,"5min":0.68999999999999995,"15min":0.53000000000000003},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.19700000000000001},"max":{"1min":23.207999999999998,"5min":23.207999999999998,"15min":23.207999999999998},"last":0.34100000000000003}]},{"osd":2,"last update":"Tue Oct 17 01:47:45 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.71299999999999997,"15min":0.57699999999999996},"min":{"1min":0.214,"5min":0.184,"15min":0.182},"max":{"1min":2.7599999999999998,"5min":13.757,"15min":13.757},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.81299999999999994,"15min":0.65800000000000003},"min":{"1min":0.28199999999999997,"5min":0.24099999999999999,"15min":0.23100000000000001},"max":{"1min":2.8079999999999998,"5min":14.195,"15min":14.195},"last":0.29799999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739159,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4869164,"kb_used_data":3489828,"kb_used_omap":289,"kb_used_meta":1379038,"kb_avail":89502676,"statfs":{"total":96636764160,"available":91650740224,"internally_reserved":0,"allocated":3573583872,"data_stored":4436067394,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":296403,"internal_metadata":1412135469},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,5,12,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:47:19 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.51500000000000001,"15min":0.55900000000000005},"min":{"1min":0.185,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.86899999999999999,"5min":4.5979999999999999,"15min":15.317},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.48399999999999999,"15min":0.55600000000000005},"min":{"1min":0.216,"5min":0.189,"15min":0.17299999999999999},"max":{"1min":1.2230000000000001,"5min":4.6609999999999996,"15min":7.5789999999999997},"last":0.46200000000000002}]},{"osd":2,"last update":"Tue Oct 17 01:47:19 2023","interfaces":[{"interface":"back","average":{"1min":0.30099999999999999,"5min":0.33600000000000002,"15min":0.38600000000000001},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.57199999999999995,"5min":1.98,"15min":5.7709999999999999},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.48899999999999999,"15min":0.57799999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.19400000000000001},"max":{"1min":3.9590000000000001,"5min":3.9590000000000001,"15min":7.6230000000000002},"last":0.49299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3573297152,"data_stored":4435894272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3465900032,"data_stored":4303732755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7035072512,"data_stored":8735502355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:47:58.732 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:47:59.196 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:47:59.197 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:47:59.234 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2068,"stamp":"2023-10-17T01:47:57.329848+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26690939736,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1747249,"num_read_kb":6795021,"num_write":1719185,"num_write_kb":6876391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81538,"ondisk_log_size":81538,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":17891004,"kb_used_data":13744612,"kb_used_omap":1134,"kb_used_meta":4145233,"kb_avail":265224516,"statfs":{"total":289910292480,"available":271589904384,"internally_reserved":0,"allocated":14074482688,"data_stored":17474468633,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1161646,"internal_metadata":4244719186},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,5,8,16,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16678912,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3875,"num_read_kb":15476,"num_write":4084,"num_write_kb":16336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002102"},"pg_stats":[{"pgid":"2.7","version":"95'11211","reported_seq":16037,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:47:55.264067+0000","last_change":"2023-10-17T00:42:12.798254+0000","last_active":"2023-10-17T01:47:55.264067+0000","last_peered":"2023-10-17T01:47:55.264067+0000","last_clean":"2023-10-17T01:47:55.264067+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:47:55.264067+0000","last_undegraded":"2023-10-17T01:47:55.264067+0000","last_fullsized":"2023-10-17T01:47:55.264067+0000","mapping_epoch":13,"log_start":"50'1200","ondisk_log_start":"50'1200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:46:37.107206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"95'8444","reported_seq":13231,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:47:55.263437+0000","last_change":"2023-10-17T00:42:12.800590+0000","last_active":"2023-10-17T01:47:55.263437+0000","last_peered":"2023-10-17T01:47:55.263437+0000","last_clean":"2023-10-17T01:47:55.263437+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:47:55.263437+0000","last_undegraded":"2023-10-17T01:47:55.263437+0000","last_fullsized":"2023-10-17T01:47:55.263437+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":8444,"log_dups_size":0,"ondisk_log_size":8444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T09:24:36.749724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"95'5661","reported_seq":8146,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:47:55.263145+0000","last_change":"2023-10-17T00:42:12.798876+0000","last_active":"2023-10-17T01:47:55.263145+0000","last_peered":"2023-10-17T01:47:55.263145+0000","last_clean":"2023-10-17T01:47:55.263145+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:47:55.263145+0000","last_undegraded":"2023-10-17T01:47:55.263145+0000","last_fullsized":"2023-10-17T01:47:55.263145+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":5661,"log_dups_size":0,"ondisk_log_size":5661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:08:55.627747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"95'8468","reported_seq":12660,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:47:55.263263+0000","last_change":"2023-10-17T00:42:12.796917+0000","last_active":"2023-10-17T01:47:55.263263+0000","last_peered":"2023-10-17T01:47:55.263263+0000","last_clean":"2023-10-17T01:47:55.263263+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:47:55.263263+0000","last_undegraded":"2023-10-17T01:47:55.263263+0000","last_fullsized":"2023-10-17T01:47:55.263263+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":8468,"log_dups_size":0,"ondisk_log_size":8468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:54:27.448734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"95'436673","reported_seq":871667,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:47:54.867674+0000","last_change":"2023-10-17T00:42:20.880933+0000","last_active":"2023-10-17T01:47:54.867674+0000","last_peered":"2023-10-17T01:47:54.867674+0000","last_clean":"2023-10-17T01:47:54.867674+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:47:54.867674+0000","last_undegraded":"2023-10-17T01:47:54.867674+0000","last_fullsized":"2023-10-17T01:47:54.867674+0000","mapping_epoch":16,"log_start":"93'426608","ondisk_log_start":"93'426608","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:20:26.021756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":6764609536,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":436146,"num_read_kb":1713784,"num_write":435375,"num_write_kb":1741500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"64'32","reported_seq":222,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-17T01:47:09.549761+0000","last_change":"2023-10-17T00:42:12.797141+0000","last_active":"2023-10-17T01:47:09.549761+0000","last_peered":"2023-10-17T01:47:09.549761+0000","last_clean":"2023-10-17T01:47:09.549761+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:47:09.549761+0000","last_undegraded":"2023-10-17T01:47:09.549761+0000","last_fullsized":"2023-10-17T01:47:09.549761+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:01:11.302607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"95'420469","reported_seq":840068,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:47:55.337291+0000","last_change":"2023-10-17T00:42:20.886926+0000","last_active":"2023-10-17T01:47:55.337291+0000","last_peered":"2023-10-17T01:47:55.337291+0000","last_clean":"2023-10-17T01:47:55.337291+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:47:55.337291+0000","last_undegraded":"2023-10-17T01:47:55.337291+0000","last_fullsized":"2023-10-17T01:47:55.337291+0000","mapping_epoch":16,"log_start":"93'410416","ondisk_log_start":"93'410416","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:20:34.646714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6506606592,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":420699,"num_read_kb":1654904,"num_write":419221,"num_write_kb":1676884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"91'61","reported_seq":2433,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:47:50.144161+0000","last_change":"2023-10-17T00:42:12.799004+0000","last_active":"2023-10-17T01:47:50.144161+0000","last_peered":"2023-10-17T01:47:50.144161+0000","last_clean":"2023-10-17T01:47:50.144161+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:47:50.144161+0000","last_undegraded":"2023-10-17T01:47:50.144161+0000","last_fullsized":"2023-10-17T01:47:50.144161+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:21:48.553692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"95'438313","reported_seq":874391,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:47:54.873723+0000","last_change":"2023-10-17T00:42:20.880768+0000","last_active":"2023-10-17T01:47:54.873723+0000","last_peered":"2023-10-17T01:47:54.873723+0000","last_clean":"2023-10-17T01:47:54.873723+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:47:54.873723+0000","last_undegraded":"2023-10-17T01:47:54.873723+0000","last_fullsized":"2023-10-17T01:47:54.873723+0000","mapping_epoch":16,"log_start":"93'428217","ondisk_log_start":"93'428217","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:38:33.949294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":6784278528,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":437235,"num_read_kb":1718856,"num_write":437012,"num_write_kb":1748048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"91'103","reported_seq":3666,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:47:50.144233+0000","last_change":"2023-10-17T00:42:12.799218+0000","last_active":"2023-10-17T01:47:50.144233+0000","last_peered":"2023-10-17T01:47:50.144233+0000","last_clean":"2023-10-17T01:47:50.144233+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:47:50.144233+0000","last_undegraded":"2023-10-17T01:47:50.144233+0000","last_fullsized":"2023-10-17T01:47:50.144233+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:00:25.745996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"95'428220","reported_seq":856297,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:47:55.337113+0000","last_change":"2023-10-17T00:42:20.888387+0000","last_active":"2023-10-17T01:47:55.337113+0000","last_peered":"2023-10-17T01:47:55.337113+0000","last_clean":"2023-10-17T01:47:55.337113+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:47:55.337113+0000","last_undegraded":"2023-10-17T01:47:55.337113+0000","last_fullsized":"2023-10-17T01:47:55.337113+0000","mapping_epoch":16,"log_start":"93'418216","ondisk_log_start":"93'418216","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:38:25.375110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6634852371,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":429207,"num_read_kb":1686824,"num_write":426947,"num_write_kb":1707782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"95'8443","reported_seq":11999,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:47:55.263816+0000","last_change":"2023-10-17T00:42:12.796559+0000","last_active":"2023-10-17T01:47:55.263816+0000","last_peered":"2023-10-17T01:47:55.263816+0000","last_clean":"2023-10-17T01:47:55.263816+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:47:55.263816+0000","last_undegraded":"2023-10-17T01:47:55.263816+0000","last_fullsized":"2023-10-17T01:47:55.263816+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":8443,"log_dups_size":0,"ondisk_log_size":8443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:59:45.094919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":253,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-17T01:46:34.868695+0000","last_change":"2023-10-17T00:42:10.044091+0000","last_active":"2023-10-17T01:46:34.868695+0000","last_peered":"2023-10-17T01:46:34.868695+0000","last_clean":"2023-10-17T01:46:34.868695+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:46:34.868695+0000","last_undegraded":"2023-10-17T01:46:34.868695+0000","last_fullsized":"2023-10-17T01:46:34.868695+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:08.753993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:08.753993+0000","last_clean_scrub_stamp":"2023-10-17T00:42:08.753993+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:18:21.220207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26690347027,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1723287,"num_read_kb":6774368,"num_write":1718555,"num_write_kb":6874214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14074269696,"data_stored":17475129382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40218,"ondisk_log_size":40218,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":41223,"ondisk_log_size":41223,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739161,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8428716,"kb_used_data":6869796,"kb_used_omap":560,"kb_used_meta":1558351,"kb_avail":85943124,"statfs":{"total":96636764160,"available":88005758976,"internally_reserved":0,"allocated":7034671104,"data_stored":8734535516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":574073,"internal_metadata":1595751815},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:46:59 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.30599999999999999,"15min":0.35999999999999999},"min":{"1min":0.183,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38100000000000001,"5min":0.93100000000000005,"15min":4.5030000000000001},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.437,"15min":0.52400000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.183},"max":{"1min":4.1959999999999997,"5min":4.1959999999999997,"15min":5.4240000000000004},"last":0.41399999999999998}]},{"osd":1,"last update":"Tue Oct 17 01:47:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.58299999999999996,"15min":0.497},"min":{"1min":0.223,"5min":0.19400000000000001,"15min":0.191},"max":{"1min":2.7799999999999998,"5min":12.413,"15min":12.413},"last":0.307},{"interface":"front","average":{"1min":0.497,"5min":0.58199999999999996,"15min":0.47899999999999998},"min":{"1min":0.26400000000000001,"5min":0.223,"15min":0.223},"max":{"1min":2.7480000000000002,"5min":5.0529999999999999,"15min":5.282},"last":0.38400000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739159,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4593124,"kb_used_data":3384988,"kb_used_omap":284,"kb_used_meta":1207843,"kb_avail":89778716,"statfs":{"total":96636764160,"available":91933405184,"internally_reserved":0,"allocated":3466227712,"data_stored":4303865723,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":291170,"internal_metadata":1236831902},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:47:45 2023","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":0.64100000000000001,"15min":0.45100000000000001},"min":{"1min":0.17799999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":23.177,"5min":23.177,"15min":23.177},"last":0.32400000000000001},{"interface":"front","average":{"1min":1.6579999999999999,"5min":0.68999999999999995,"15min":0.53000000000000003},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.19700000000000001},"max":{"1min":23.207999999999998,"5min":23.207999999999998,"15min":23.207999999999998},"last":0.34100000000000003}]},{"osd":2,"last update":"Tue Oct 17 01:47:45 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.71299999999999997,"15min":0.57699999999999996},"min":{"1min":0.214,"5min":0.184,"15min":0.182},"max":{"1min":2.7599999999999998,"5min":13.757,"15min":13.757},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.81299999999999994,"15min":0.65800000000000003},"min":{"1min":0.28199999999999997,"5min":0.24099999999999999,"15min":0.23100000000000001},"max":{"1min":2.8079999999999998,"5min":14.195,"15min":14.195},"last":0.29799999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739159,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4869164,"kb_used_data":3489828,"kb_used_omap":289,"kb_used_meta":1379038,"kb_avail":89502676,"statfs":{"total":96636764160,"available":91650740224,"internally_reserved":0,"allocated":3573583872,"data_stored":4436067394,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":296403,"internal_metadata":1412135469},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,5,12,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:47:19 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.51500000000000001,"15min":0.55900000000000005},"min":{"1min":0.185,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.86899999999999999,"5min":4.5979999999999999,"15min":15.317},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.48399999999999999,"15min":0.55600000000000005},"min":{"1min":0.216,"5min":0.189,"15min":0.17299999999999999},"max":{"1min":1.2230000000000001,"5min":4.6609999999999996,"15min":7.5789999999999997},"last":0.46200000000000002}]},{"osd":2,"last update":"Tue Oct 17 01:47:19 2023","interfaces":[{"interface":"back","average":{"1min":0.30099999999999999,"5min":0.33600000000000002,"15min":0.38600000000000001},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.57199999999999995,"5min":1.98,"15min":5.7709999999999999},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.48899999999999999,"15min":0.57799999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.19400000000000001},"max":{"1min":3.9590000000000001,"5min":3.9590000000000001,"15min":7.6230000000000002},"last":0.49299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3573297152,"data_stored":4435894272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3465900032,"data_stored":4303732755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7035072512,"data_stored":8735502355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:47:59.236 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-17T01:47:59.236 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:47:59.736 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:47:59.737 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:47:59.765 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2069,"stamp":"2023-10-17T01:47:59.330199+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26690939736,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1747249,"num_read_kb":6795021,"num_write":1719185,"num_write_kb":6876391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81538,"ondisk_log_size":81538,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":17908944,"kb_used_data":13752824,"kb_used_omap":1134,"kb_used_meta":4154961,"kb_avail":265206576,"statfs":{"total":289910292480,"available":271571533824,"internally_reserved":0,"allocated":14082891776,"data_stored":17482877721,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1161646,"internal_metadata":4254680658},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,6,5,9,20,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16678912,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3875,"num_read_kb":15476,"num_write":4084,"num_write_kb":16336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002025"},"pg_stats":[{"pgid":"2.7","version":"95'11211","reported_seq":16037,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:47:55.264067+0000","last_change":"2023-10-17T00:42:12.798254+0000","last_active":"2023-10-17T01:47:55.264067+0000","last_peered":"2023-10-17T01:47:55.264067+0000","last_clean":"2023-10-17T01:47:55.264067+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:47:55.264067+0000","last_undegraded":"2023-10-17T01:47:55.264067+0000","last_fullsized":"2023-10-17T01:47:55.264067+0000","mapping_epoch":13,"log_start":"50'1200","ondisk_log_start":"50'1200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:46:37.107206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"95'8444","reported_seq":13231,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:47:55.263437+0000","last_change":"2023-10-17T00:42:12.800590+0000","last_active":"2023-10-17T01:47:55.263437+0000","last_peered":"2023-10-17T01:47:55.263437+0000","last_clean":"2023-10-17T01:47:55.263437+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:47:55.263437+0000","last_undegraded":"2023-10-17T01:47:55.263437+0000","last_fullsized":"2023-10-17T01:47:55.263437+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":8444,"log_dups_size":0,"ondisk_log_size":8444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T09:24:36.749724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"95'5661","reported_seq":8146,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:47:55.263145+0000","last_change":"2023-10-17T00:42:12.798876+0000","last_active":"2023-10-17T01:47:55.263145+0000","last_peered":"2023-10-17T01:47:55.263145+0000","last_clean":"2023-10-17T01:47:55.263145+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:47:55.263145+0000","last_undegraded":"2023-10-17T01:47:55.263145+0000","last_fullsized":"2023-10-17T01:47:55.263145+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":5661,"log_dups_size":0,"ondisk_log_size":5661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:08:55.627747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"95'8468","reported_seq":12660,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:47:55.263263+0000","last_change":"2023-10-17T00:42:12.796917+0000","last_active":"2023-10-17T01:47:55.263263+0000","last_peered":"2023-10-17T01:47:55.263263+0000","last_clean":"2023-10-17T01:47:55.263263+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:47:55.263263+0000","last_undegraded":"2023-10-17T01:47:55.263263+0000","last_fullsized":"2023-10-17T01:47:55.263263+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":8468,"log_dups_size":0,"ondisk_log_size":8468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:54:27.448734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"95'436673","reported_seq":871667,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:47:54.867674+0000","last_change":"2023-10-17T00:42:20.880933+0000","last_active":"2023-10-17T01:47:54.867674+0000","last_peered":"2023-10-17T01:47:54.867674+0000","last_clean":"2023-10-17T01:47:54.867674+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:47:54.867674+0000","last_undegraded":"2023-10-17T01:47:54.867674+0000","last_fullsized":"2023-10-17T01:47:54.867674+0000","mapping_epoch":16,"log_start":"93'426608","ondisk_log_start":"93'426608","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:20:26.021756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":6764609536,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":436146,"num_read_kb":1713784,"num_write":435375,"num_write_kb":1741500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"64'32","reported_seq":222,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-17T01:47:09.549761+0000","last_change":"2023-10-17T00:42:12.797141+0000","last_active":"2023-10-17T01:47:09.549761+0000","last_peered":"2023-10-17T01:47:09.549761+0000","last_clean":"2023-10-17T01:47:09.549761+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:47:09.549761+0000","last_undegraded":"2023-10-17T01:47:09.549761+0000","last_fullsized":"2023-10-17T01:47:09.549761+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:01:11.302607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"95'420469","reported_seq":840068,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:47:55.337291+0000","last_change":"2023-10-17T00:42:20.886926+0000","last_active":"2023-10-17T01:47:55.337291+0000","last_peered":"2023-10-17T01:47:55.337291+0000","last_clean":"2023-10-17T01:47:55.337291+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:47:55.337291+0000","last_undegraded":"2023-10-17T01:47:55.337291+0000","last_fullsized":"2023-10-17T01:47:55.337291+0000","mapping_epoch":16,"log_start":"93'410416","ondisk_log_start":"93'410416","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:20:34.646714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6506606592,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":420699,"num_read_kb":1654904,"num_write":419221,"num_write_kb":1676884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"91'61","reported_seq":2435,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:47:55.263834+0000","last_change":"2023-10-17T00:42:12.799004+0000","last_active":"2023-10-17T01:47:55.263834+0000","last_peered":"2023-10-17T01:47:55.263834+0000","last_clean":"2023-10-17T01:47:55.263834+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:47:55.263834+0000","last_undegraded":"2023-10-17T01:47:55.263834+0000","last_fullsized":"2023-10-17T01:47:55.263834+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:21:48.553692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"95'438313","reported_seq":874391,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:47:54.873723+0000","last_change":"2023-10-17T00:42:20.880768+0000","last_active":"2023-10-17T01:47:54.873723+0000","last_peered":"2023-10-17T01:47:54.873723+0000","last_clean":"2023-10-17T01:47:54.873723+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:47:54.873723+0000","last_undegraded":"2023-10-17T01:47:54.873723+0000","last_fullsized":"2023-10-17T01:47:54.873723+0000","mapping_epoch":16,"log_start":"93'428217","ondisk_log_start":"93'428217","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:38:33.949294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":6784278528,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":437235,"num_read_kb":1718856,"num_write":437012,"num_write_kb":1748048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"91'103","reported_seq":3669,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:47:55.268870+0000","last_change":"2023-10-17T00:42:12.799218+0000","last_active":"2023-10-17T01:47:55.268870+0000","last_peered":"2023-10-17T01:47:55.268870+0000","last_clean":"2023-10-17T01:47:55.268870+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:47:55.268870+0000","last_undegraded":"2023-10-17T01:47:55.268870+0000","last_fullsized":"2023-10-17T01:47:55.268870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:00:25.745996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"95'428220","reported_seq":856297,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:47:55.337113+0000","last_change":"2023-10-17T00:42:20.888387+0000","last_active":"2023-10-17T01:47:55.337113+0000","last_peered":"2023-10-17T01:47:55.337113+0000","last_clean":"2023-10-17T01:47:55.337113+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:47:55.337113+0000","last_undegraded":"2023-10-17T01:47:55.337113+0000","last_fullsized":"2023-10-17T01:47:55.337113+0000","mapping_epoch":16,"log_start":"93'418216","ondisk_log_start":"93'418216","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:38:25.375110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6634852371,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":429207,"num_read_kb":1686824,"num_write":426947,"num_write_kb":1707782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"95'8443","reported_seq":11999,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:47:55.263816+0000","last_change":"2023-10-17T00:42:12.796559+0000","last_active":"2023-10-17T01:47:55.263816+0000","last_peered":"2023-10-17T01:47:55.263816+0000","last_clean":"2023-10-17T01:47:55.263816+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:47:55.263816+0000","last_undegraded":"2023-10-17T01:47:55.263816+0000","last_fullsized":"2023-10-17T01:47:55.263816+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":8443,"log_dups_size":0,"ondisk_log_size":8443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:59:45.094919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":253,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-17T01:46:34.868695+0000","last_change":"2023-10-17T00:42:10.044091+0000","last_active":"2023-10-17T01:46:34.868695+0000","last_peered":"2023-10-17T01:46:34.868695+0000","last_clean":"2023-10-17T01:46:34.868695+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:46:34.868695+0000","last_undegraded":"2023-10-17T01:46:34.868695+0000","last_fullsized":"2023-10-17T01:46:34.868695+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:08.753993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:08.753993+0000","last_clean_scrub_stamp":"2023-10-17T00:42:08.753993+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:18:21.220207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26690347027,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1723287,"num_read_kb":6774368,"num_write":1718555,"num_write_kb":6874214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14082707456,"data_stored":17483567142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40218,"ondisk_log_size":40218,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":41223,"ondisk_log_size":41223,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739162,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8446656,"kb_used_data":6878008,"kb_used_omap":560,"kb_used_meta":1568079,"kb_avail":85925184,"statfs":{"total":96636764160,"available":87987388416,"internally_reserved":0,"allocated":7043080192,"data_stored":8742944604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":574073,"internal_metadata":1605713287},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,3,4,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:46:59 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.30599999999999999,"15min":0.35999999999999999},"min":{"1min":0.183,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38100000000000001,"5min":0.93100000000000005,"15min":4.5030000000000001},"last":0.311},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.437,"15min":0.52400000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.183},"max":{"1min":4.1959999999999997,"5min":4.1959999999999997,"15min":5.4240000000000004},"last":0.34599999999999997}]},{"osd":1,"last update":"Tue Oct 17 01:47:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.58299999999999996,"15min":0.497},"min":{"1min":0.223,"5min":0.19400000000000001,"15min":0.191},"max":{"1min":2.7799999999999998,"5min":12.413,"15min":12.413},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.497,"5min":0.58199999999999996,"15min":0.47899999999999998},"min":{"1min":0.26400000000000001,"5min":0.223,"15min":0.223},"max":{"1min":2.7480000000000002,"5min":5.0529999999999999,"15min":5.282},"last":0.46100000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739159,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4593124,"kb_used_data":3384988,"kb_used_omap":284,"kb_used_meta":1207843,"kb_avail":89778716,"statfs":{"total":96636764160,"available":91933405184,"internally_reserved":0,"allocated":3466227712,"data_stored":4303865723,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":291170,"internal_metadata":1236831902},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:47:45 2023","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":0.64100000000000001,"15min":0.45100000000000001},"min":{"1min":0.17799999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":23.177,"5min":23.177,"15min":23.177},"last":0.32400000000000001},{"interface":"front","average":{"1min":1.6579999999999999,"5min":0.68999999999999995,"15min":0.53000000000000003},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.19700000000000001},"max":{"1min":23.207999999999998,"5min":23.207999999999998,"15min":23.207999999999998},"last":0.34100000000000003}]},{"osd":2,"last update":"Tue Oct 17 01:47:45 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.71299999999999997,"15min":0.57699999999999996},"min":{"1min":0.214,"5min":0.184,"15min":0.182},"max":{"1min":2.7599999999999998,"5min":13.757,"15min":13.757},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.81299999999999994,"15min":0.65800000000000003},"min":{"1min":0.28199999999999997,"5min":0.24099999999999999,"15min":0.23100000000000001},"max":{"1min":2.8079999999999998,"5min":14.195,"15min":14.195},"last":0.29799999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739159,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4869164,"kb_used_data":3489828,"kb_used_omap":289,"kb_used_meta":1379038,"kb_avail":89502676,"statfs":{"total":96636764160,"available":91650740224,"internally_reserved":0,"allocated":3573583872,"data_stored":4436067394,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":296403,"internal_metadata":1412135469},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,5,12,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:47:19 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.51500000000000001,"15min":0.55900000000000005},"min":{"1min":0.185,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.86899999999999999,"5min":4.5979999999999999,"15min":15.317},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.48399999999999999,"15min":0.55600000000000005},"min":{"1min":0.216,"5min":0.189,"15min":0.17299999999999999},"max":{"1min":1.2230000000000001,"5min":4.6609999999999996,"15min":7.5789999999999997},"last":0.46200000000000002}]},{"osd":2,"last update":"Tue Oct 17 01:47:19 2023","interfaces":[{"interface":"back","average":{"1min":0.30099999999999999,"5min":0.33600000000000002,"15min":0.38600000000000001},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.57199999999999995,"5min":1.98,"15min":5.7709999999999999},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.48899999999999999,"15min":0.57799999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.19400000000000001},"max":{"1min":3.9590000000000001,"5min":3.9590000000000001,"15min":7.6230000000000002},"last":0.49299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3573297152,"data_stored":4435894272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3465900032,"data_stored":4303732755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7043510272,"data_stored":8743940115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:47:59.767 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-17T01:48:00.450 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:48:00.451 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":95,"fsid":"ce1c5384-f13f-4ca6-8231-3f754aa73d4f","created":"2023-10-17T00:41:55.707833+0000","modified":"2023-10-17T01:47:42.009000+0000","last_up_change":"2023-10-17T00:42:06.746971+0000","last_in_change":"2023-10-17T00:41:56.626469+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-17T00:42:08.546652+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-17T00:42:10.920165+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2023-10-17T00:42:16.850534+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":43,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"796660b8-af5d-47da-94d6-8e79f75895f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":2140921368},{"type":"v1","addr":"172.21.15.169:6817","nonce":2140921368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":2140921368},{"type":"v1","addr":"172.21.15.169:6819","nonce":2140921368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":2140921368},{"type":"v1","addr":"172.21.15.169:6823","nonce":2140921368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":2140921368},{"type":"v1","addr":"172.21.15.169:6821","nonce":2140921368}]},"public_addr":"172.21.15.169:6817/2140921368","cluster_addr":"172.21.15.169:6819/2140921368","heartbeat_back_addr":"172.21.15.169:6823/2140921368","heartbeat_front_addr":"172.21.15.169:6821/2140921368","state":["exists","up"]},{"osd":1,"uuid":"f4c91e4d-51e9-4f95-a8c9-407c4d88298f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":516036749},{"type":"v1","addr":"172.21.15.169:6801","nonce":516036749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":516036749},{"type":"v1","addr":"172.21.15.169:6805","nonce":516036749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":516036749},{"type":"v1","addr":"172.21.15.169:6815","nonce":516036749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":516036749},{"type":"v1","addr":"172.21.15.169:6810","nonce":516036749}]},"public_addr":"172.21.15.169:6801/516036749","cluster_addr":"172.21.15.169:6805/516036749","heartbeat_back_addr":"172.21.15.169:6815/516036749","heartbeat_front_addr":"172.21.15.169:6810/516036749","state":["exists","up"]},{"osd":2,"uuid":"f5e0fd93-fa0b-4c12-9996-f3e959ded07a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":1146691734},{"type":"v1","addr":"172.21.15.169:6803","nonce":1146691734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":1146691734},{"type":"v1","addr":"172.21.15.169:6807","nonce":1146691734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6813","nonce":1146691734},{"type":"v1","addr":"172.21.15.169:6814","nonce":1146691734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":1146691734},{"type":"v1","addr":"172.21.15.169:6811","nonce":1146691734}]},"public_addr":"172.21.15.169:6803/1146691734","cluster_addr":"172.21.15.169:6807/1146691734","heartbeat_back_addr":"172.21.15.169:6814/1146691734","heartbeat_front_addr":"172.21.15.169:6811/1146691734","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-17T00:42:02.258086+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-17T00:42:02.154164+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-17T00:42:02.110190+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:0/1398705567":"2023-10-17T02:11:13.663419+0000","172.21.15.169:0/3137532849":"2023-10-17T02:08:45.072587+0000","172.21.15.169:0/4244069168":"2023-10-17T02:07:35.907569+0000","172.21.15.169:0/1275570382":"2023-10-17T02:06:27.966296+0000","172.21.15.169:0/2965545755":"2023-10-17T02:00:33.522008+0000","172.21.15.169:0/1004139816":"2023-10-17T01:59:24.187073+0000","172.21.15.169:0/1523367413":"2023-10-17T01:57:07.524166+0000","172.21.15.169:0/1405644418":"2023-10-17T02:05:19.426753+0000","172.21.15.169:0/315948540":"2023-10-17T02:02:50.888913+0000","172.21.15.169:0/2848467669":"2023-10-17T01:48:48.678225+0000","172.21.15.169:0/471946586":"2023-10-17T02:01:53.466046+0000","172.21.15.169:0/715849107":"2023-10-17T01:53:32.862390+0000","172.21.15.169:0/2992472699":"2023-10-17T02:03:59.451290+0000","172.21.15.169:0/3869173669":"2023-10-17T01:49:57.023624+0000","172.21.15.169:0/1704238022":"2023-10-17T01:51:05.812666+0000","172.21.15.169:0/551423965":"2023-10-17T02:09:53.821528+0000","172.21.15.169:0/144294311":"2023-10-17T01:55:59.930831+0000","172.21.15.169:0/895333782":"2023-10-17T01:52:24.494940+0000","172.21.15.169:0/2169492734":"2023-10-17T01:58:15.787026+0000","172.21.15.169:0/1212236727":"2023-10-17T01:54:40.692891+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-17T01:48:01.503 INFO:tasks.ceph:Scrubbing osd.0 2023-10-17T01:48:01.504 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-17T01:48:01.759 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2023-10-17T01:48:01.760 INFO:teuthology.orchestra.run.smithi169.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-17T01:48:01.760 INFO:teuthology.orchestra.run.smithi169.stdout:} 2023-10-17T01:48:01.815 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-17T01:48:02.306 INFO:teuthology.orchestra.run.smithi169.stderr:instructed osd(s) 0 to deep-scrub 2023-10-17T01:48:02.343 INFO:tasks.ceph:Scrubbing osd.1 2023-10-17T01:48:02.343 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-17T01:48:02.589 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2023-10-17T01:48:02.590 INFO:teuthology.orchestra.run.smithi169.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-17T01:48:02.590 INFO:teuthology.orchestra.run.smithi169.stdout:} 2023-10-17T01:48:02.644 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-17T01:48:03.180 INFO:teuthology.orchestra.run.smithi169.stderr:instructed osd(s) 1 to deep-scrub 2023-10-17T01:48:03.214 INFO:tasks.ceph:Scrubbing osd.2 2023-10-17T01:48:03.215 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-17T01:48:03.518 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2023-10-17T01:48:03.518 INFO:teuthology.orchestra.run.smithi169.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-17T01:48:03.519 INFO:teuthology.orchestra.run.smithi169.stdout:} 2023-10-17T01:48:03.548 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-17T01:48:03.758 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:48:03.749+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:48:03.770 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:48:03.761+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:48:03.780 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:48:03.777+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:48:04.038 INFO:teuthology.orchestra.run.smithi169.stderr:instructed osd(s) 2 to deep-scrub 2023-10-17T01:48:04.075 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:48:04.534 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:48:04.534 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:48:04.570 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2071,"stamp":"2023-10-17T01:48:03.333769+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26698877784,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1749136,"num_read_kb":6802553,"num_write":1721121,"num_write_kb":6884135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81589,"ondisk_log_size":81589,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":17995272,"kb_used_data":13768368,"kb_used_omap":1134,"kb_used_meta":4225745,"kb_avail":265120248,"statfs":{"total":289910292480,"available":271483133952,"internally_reserved":0,"allocated":14098808832,"data_stored":17498794777,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1161646,"internal_metadata":4327163474},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3,17,15,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16326656,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3885,"num_read_kb":15516,"num_write":3991,"num_write_kb":15964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004998"},"pg_stats":[{"pgid":"2.7","version":"95'11215","reported_seq":16045,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:48:00.384033+0000","last_change":"2023-10-17T00:42:12.798254+0000","last_active":"2023-10-17T01:48:00.384033+0000","last_peered":"2023-10-17T01:48:00.384033+0000","last_clean":"2023-10-17T01:48:00.384033+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:48:00.384033+0000","last_undegraded":"2023-10-17T01:48:00.384033+0000","last_fullsized":"2023-10-17T01:48:00.384033+0000","mapping_epoch":13,"log_start":"50'1200","ondisk_log_start":"50'1200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:46:37.107206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"95'8447","reported_seq":13237,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:48:00.383627+0000","last_change":"2023-10-17T00:42:12.800590+0000","last_active":"2023-10-17T01:48:00.383627+0000","last_peered":"2023-10-17T01:48:00.383627+0000","last_clean":"2023-10-17T01:48:00.383627+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:48:00.383627+0000","last_undegraded":"2023-10-17T01:48:00.383627+0000","last_fullsized":"2023-10-17T01:48:00.383627+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":8447,"log_dups_size":0,"ondisk_log_size":8447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T09:24:36.749724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"95'5663","reported_seq":8150,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:48:00.383149+0000","last_change":"2023-10-17T00:42:12.798876+0000","last_active":"2023-10-17T01:48:00.383149+0000","last_peered":"2023-10-17T01:48:00.383149+0000","last_clean":"2023-10-17T01:48:00.383149+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:48:00.383149+0000","last_undegraded":"2023-10-17T01:48:00.383149+0000","last_fullsized":"2023-10-17T01:48:00.383149+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":5663,"log_dups_size":0,"ondisk_log_size":5663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:08:55.627747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"95'8471","reported_seq":12666,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:48:00.383307+0000","last_change":"2023-10-17T00:42:12.796917+0000","last_active":"2023-10-17T01:48:00.383307+0000","last_peered":"2023-10-17T01:48:00.383307+0000","last_clean":"2023-10-17T01:48:00.383307+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:48:00.383307+0000","last_undegraded":"2023-10-17T01:48:00.383307+0000","last_fullsized":"2023-10-17T01:48:00.383307+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":8471,"log_dups_size":0,"ondisk_log_size":8471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:54:27.448734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"95'437134","reported_seq":872590,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:47:59.876531+0000","last_change":"2023-10-17T00:42:20.880933+0000","last_active":"2023-10-17T01:47:59.876531+0000","last_peered":"2023-10-17T01:47:59.876531+0000","last_clean":"2023-10-17T01:47:59.876531+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:47:59.876531+0000","last_undegraded":"2023-10-17T01:47:59.876531+0000","last_fullsized":"2023-10-17T01:47:59.876531+0000","mapping_epoch":16,"log_start":"93'427108","ondisk_log_start":"93'427108","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:20:26.021756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":6766534656,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":436608,"num_read_kb":1715628,"num_write":435836,"num_write_kb":1743344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"64'32","reported_seq":222,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-17T01:47:09.549761+0000","last_change":"2023-10-17T00:42:12.797141+0000","last_active":"2023-10-17T01:47:09.549761+0000","last_peered":"2023-10-17T01:47:09.549761+0000","last_clean":"2023-10-17T01:47:09.549761+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:47:09.549761+0000","last_undegraded":"2023-10-17T01:47:09.549761+0000","last_fullsized":"2023-10-17T01:47:09.549761+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:01:11.302607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"95'420975","reported_seq":840999,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:48:00.348773+0000","last_change":"2023-10-17T00:42:20.886926+0000","last_active":"2023-10-17T01:48:00.348773+0000","last_peered":"2023-10-17T01:48:00.348773+0000","last_clean":"2023-10-17T01:48:00.348773+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:48:00.348773+0000","last_undegraded":"2023-10-17T01:48:00.348773+0000","last_fullsized":"2023-10-17T01:48:00.348773+0000","mapping_epoch":16,"log_start":"93'410916","ondisk_log_start":"93'410916","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:20:34.646714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6508670976,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":421125,"num_read_kb":1656608,"num_write":419727,"num_write_kb":1678908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"91'61","reported_seq":2437,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:48:00.383360+0000","last_change":"2023-10-17T00:42:12.799004+0000","last_active":"2023-10-17T01:48:00.383360+0000","last_peered":"2023-10-17T01:48:00.383360+0000","last_clean":"2023-10-17T01:48:00.383360+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:48:00.383360+0000","last_undegraded":"2023-10-17T01:48:00.383360+0000","last_fullsized":"2023-10-17T01:48:00.383360+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:21:48.553692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"95'438807","reported_seq":875407,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:47:59.878738+0000","last_change":"2023-10-17T00:42:20.880768+0000","last_active":"2023-10-17T01:47:59.878738+0000","last_peered":"2023-10-17T01:47:59.878738+0000","last_clean":"2023-10-17T01:47:59.878738+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:47:59.878738+0000","last_undegraded":"2023-10-17T01:47:59.878738+0000","last_fullsized":"2023-10-17T01:47:59.878738+0000","mapping_epoch":16,"log_start":"93'428717","ondisk_log_start":"93'428717","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:38:33.949294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":6786293760,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":437761,"num_read_kb":1720956,"num_write":437506,"num_write_kb":1750024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"91'103","reported_seq":3672,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:48:00.465420+0000","last_change":"2023-10-17T00:42:12.799218+0000","last_active":"2023-10-17T01:48:00.465420+0000","last_peered":"2023-10-17T01:48:00.465420+0000","last_clean":"2023-10-17T01:48:00.465420+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:48:00.465420+0000","last_undegraded":"2023-10-17T01:48:00.465420+0000","last_fullsized":"2023-10-17T01:48:00.465420+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:00:25.745996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"95'428695","reported_seq":857246,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:48:00.367850+0000","last_change":"2023-10-17T00:42:20.888387+0000","last_active":"2023-10-17T01:48:00.367850+0000","last_peered":"2023-10-17T01:48:00.367850+0000","last_clean":"2023-10-17T01:48:00.367850+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:48:00.367850+0000","last_undegraded":"2023-10-17T01:48:00.367850+0000","last_fullsized":"2023-10-17T01:48:00.367850+0000","mapping_epoch":16,"log_start":"93'418616","ondisk_log_start":"93'418616","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:38:25.375110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6636785683,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":429680,"num_read_kb":1688708,"num_write":427422,"num_write_kb":1709682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"95'8446","reported_seq":12005,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:48:00.383366+0000","last_change":"2023-10-17T00:42:12.796559+0000","last_active":"2023-10-17T01:48:00.383366+0000","last_peered":"2023-10-17T01:48:00.383366+0000","last_clean":"2023-10-17T01:48:00.383366+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:48:00.383366+0000","last_undegraded":"2023-10-17T01:48:00.383366+0000","last_fullsized":"2023-10-17T01:48:00.383366+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:11.776153+0000","last_clean_scrub_stamp":"2023-10-17T00:42:11.776153+0000","objects_scrubbed":0,"log_size":8446,"log_dups_size":0,"ondisk_log_size":8446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:59:45.094919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":253,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-17T01:46:34.868695+0000","last_change":"2023-10-17T00:42:10.044091+0000","last_active":"2023-10-17T01:46:34.868695+0000","last_peered":"2023-10-17T01:46:34.868695+0000","last_clean":"2023-10-17T01:46:34.868695+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:46:34.868695+0000","last_undegraded":"2023-10-17T01:46:34.868695+0000","last_fullsized":"2023-10-17T01:46:34.868695+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:08.753993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:08.753993+0000","last_clean_scrub_stamp":"2023-10-17T00:42:08.753993+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:18:21.220207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26698285075,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1725174,"num_read_kb":6781900,"num_write":1720491,"num_write_kb":6881958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14098419712,"data_stored":17499279398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40254,"ondisk_log_size":40254,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":41238,"ondisk_log_size":41238,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739163,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8480596,"kb_used_data":6885580,"kb_used_omap":560,"kb_used_meta":1594447,"kb_avail":85891244,"statfs":{"total":96636764160,"available":87952633856,"internally_reserved":0,"allocated":7050833920,"data_stored":8750698332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":574073,"internal_metadata":1632714119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:46:59 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.30599999999999999,"15min":0.35999999999999999},"min":{"1min":0.183,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38100000000000001,"5min":0.93100000000000005,"15min":4.5030000000000001},"last":0.216},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.437,"15min":0.52400000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.183},"max":{"1min":4.1959999999999997,"5min":4.1959999999999997,"15min":5.4240000000000004},"last":4.2610000000000001}]},{"osd":1,"last update":"Tue Oct 17 01:47:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.58299999999999996,"15min":0.497},"min":{"1min":0.223,"5min":0.19400000000000001,"15min":0.191},"max":{"1min":2.7799999999999998,"5min":12.413,"15min":12.413},"last":4.3419999999999996},{"interface":"front","average":{"1min":0.497,"5min":0.58199999999999996,"15min":0.47899999999999998},"min":{"1min":0.26400000000000001,"5min":0.223,"15min":0.223},"max":{"1min":2.7480000000000002,"5min":5.0529999999999999,"15min":5.282},"last":4.1790000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739160,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4619424,"kb_used_data":3389080,"kb_used_omap":284,"kb_used_meta":1230051,"kb_avail":89752416,"statfs":{"total":96636764160,"available":91906473984,"internally_reserved":0,"allocated":3470417920,"data_stored":4308055931,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":291170,"internal_metadata":1259572894},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,5,2,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:47:45 2023","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":0.64100000000000001,"15min":0.45100000000000001},"min":{"1min":0.17799999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":23.177,"5min":23.177,"15min":23.177},"last":0.309},{"interface":"front","average":{"1min":1.6579999999999999,"5min":0.68999999999999995,"15min":0.53000000000000003},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.19700000000000001},"max":{"1min":23.207999999999998,"5min":23.207999999999998,"15min":23.207999999999998},"last":0.33900000000000002}]},{"osd":2,"last update":"Tue Oct 17 01:47:45 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.71299999999999997,"15min":0.57699999999999996},"min":{"1min":0.214,"5min":0.184,"15min":0.182},"max":{"1min":2.7599999999999998,"5min":13.757,"15min":13.757},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.81299999999999994,"15min":0.65800000000000003},"min":{"1min":0.28199999999999997,"5min":0.24099999999999999,"15min":0.23100000000000001},"max":{"1min":2.8079999999999998,"5min":14.195,"15min":14.195},"last":0.29299999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739160,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4895252,"kb_used_data":3493708,"kb_used_omap":289,"kb_used_meta":1401246,"kb_avail":89476588,"statfs":{"total":96636764160,"available":91624026112,"internally_reserved":0,"allocated":3577556992,"data_stored":4440040514,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":296403,"internal_metadata":1434876461},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,10,11,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:47:19 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.51500000000000001,"15min":0.55900000000000005},"min":{"1min":0.185,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.86899999999999999,"5min":4.5979999999999999,"15min":15.317},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.48399999999999999,"15min":0.55600000000000005},"min":{"1min":0.216,"5min":0.189,"15min":0.17299999999999999},"max":{"1min":1.2230000000000001,"5min":4.6609999999999996,"15min":7.5789999999999997},"last":5.5039999999999996}]},{"osd":2,"last update":"Tue Oct 17 01:47:19 2023","interfaces":[{"interface":"back","average":{"1min":0.30099999999999999,"5min":0.33600000000000002,"15min":0.38600000000000001},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.57199999999999995,"5min":1.98,"15min":5.7709999999999999},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.48899999999999999,"15min":0.57799999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.19400000000000001},"max":{"1min":3.9590000000000001,"5min":3.9590000000000001,"15min":7.6230000000000002},"last":3.968}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3577217024,"data_stored":4439814144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3469950976,"data_stored":4307783699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7051251712,"data_stored":8751681555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:48:04.573 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-17T00:42:11.776153+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=11, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:48:04.574 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-17T00:42:11.776153+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=11, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:48:04.574 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-17T00:42:11.776153+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=11, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:48:04.575 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-17T00:42:11.776153+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=11, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:48:04.575 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:48:04.575 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-17T00:42:11.776153+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=11, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:48:04.576 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:48:04.576 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-17T00:42:11.776153+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=11, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:48:04.577 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:48:04.577 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-17T00:42:11.776153+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=11, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:48:04.577 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:48:04.578 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-17T00:42:11.776153+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=11, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:48:04.578 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-17T00:42:08.753993+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=8, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:48:04.579 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T01:48:24.579 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:48:25.131 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:48:25.132 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:48:25.167 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2081,"stamp":"2023-10-17T01:48:23.337375+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26730810200,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1756324,"num_read_kb":6831257,"num_write":1728930,"num_write_kb":6915371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81786,"ondisk_log_size":81786,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18349104,"kb_used_data":13832536,"kb_used_omap":1134,"kb_used_meta":4515409,"kb_avail":264766416,"statfs":{"total":289910292480,"available":271120809984,"internally_reserved":0,"allocated":14164516864,"data_stored":17564502809,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1161646,"internal_metadata":4623779410},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,9,17,20,23,9],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15884288,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3506,"num_read_kb":14000,"num_write":3885,"num_write_kb":15540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10746,"num_omap_keys":270,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002263"},"pg_stats":[{"pgid":"2.7","version":"95'11246","reported_seq":16097,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:48:19.607481+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T01:48:19.607481+0000","last_peered":"2023-10-17T01:48:19.607481+0000","last_clean":"2023-10-17T01:48:19.607481+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:48:19.607481+0000","last_undegraded":"2023-10-17T01:48:19.607481+0000","last_fullsized":"2023-10-17T01:48:19.607481+0000","mapping_epoch":13,"log_start":"50'1200","ondisk_log_start":"50'1200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"95'8473","reported_seq":13281,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:48:18.640119+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T01:48:18.640119+0000","last_peered":"2023-10-17T01:48:18.640119+0000","last_clean":"2023-10-17T01:48:18.640119+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:48:18.640119+0000","last_undegraded":"2023-10-17T01:48:18.640119+0000","last_fullsized":"2023-10-17T01:48:18.640119+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":8473,"log_dups_size":0,"ondisk_log_size":8473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"95'5681","reported_seq":8183,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:48:18.526795+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T01:48:18.526795+0000","last_peered":"2023-10-17T01:48:18.526795+0000","last_clean":"2023-10-17T01:48:18.526795+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:48:18.526795+0000","last_undegraded":"2023-10-17T01:48:18.526795+0000","last_fullsized":"2023-10-17T01:48:18.526795+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":5681,"log_dups_size":0,"ondisk_log_size":5681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"95'8502","reported_seq":12715,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:48:18.532239+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T01:48:18.532239+0000","last_peered":"2023-10-17T01:48:18.532239+0000","last_clean":"2023-10-17T01:48:18.532239+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:48:18.532239+0000","last_undegraded":"2023-10-17T01:48:18.532239+0000","last_fullsized":"2023-10-17T01:48:18.532239+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":8502,"log_dups_size":0,"ondisk_log_size":8502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"95'439090","reported_seq":876377,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:48:20.058870+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T01:48:20.058870+0000","last_peered":"2023-10-17T01:48:20.058870+0000","last_clean":"2023-10-17T01:48:20.058870+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:48:20.058870+0000","last_undegraded":"2023-10-17T01:48:20.058870+0000","last_fullsized":"2023-10-17T01:48:20.058870+0000","mapping_epoch":16,"log_start":"94'429008","ondisk_log_start":"94'429008","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":14,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 6s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":6774484992,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":438435,"num_read_kb":1722928,"num_write":437784,"num_write_kb":1751136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":244,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:48:03.009517+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:48:03.009517+0000","last_peered":"2023-10-17T01:48:03.009517+0000","last_clean":"2023-10-17T01:48:03.009517+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:48:03.009517+0000","last_undegraded":"2023-10-17T01:48:03.009517+0000","last_fullsized":"2023-10-17T01:48:03.009517+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"95'422867","reported_seq":844645,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:48:20.550360+0000","last_change":"2023-10-17T00:42:20.886926+0000","last_active":"2023-10-17T01:48:20.550360+0000","last_peered":"2023-10-17T01:48:20.550360+0000","last_clean":"2023-10-17T01:48:20.550360+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:48:20.550360+0000","last_undegraded":"2023-10-17T01:48:20.550360+0000","last_fullsized":"2023-10-17T01:48:20.550360+0000","mapping_epoch":16,"log_start":"94'412816","ondisk_log_start":"94'412816","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6516400128,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":422879,"num_read_kb":1663620,"num_write":421619,"num_write_kb":1686476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2463,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:48:20.868609+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T01:48:20.868609+0000","last_peered":"2023-10-17T01:48:20.868609+0000","last_clean":"2023-10-17T01:48:20.868609+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:48:20.868609+0000","last_undegraded":"2023-10-17T01:48:20.868609+0000","last_fullsized":"2023-10-17T01:48:20.868609+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"95'440803","reported_seq":879290,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:48:20.041850+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T01:48:20.041850+0000","last_peered":"2023-10-17T01:48:20.041850+0000","last_clean":"2023-10-17T01:48:20.041850+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:48:20.041850+0000","last_undegraded":"2023-10-17T01:48:20.041850+0000","last_fullsized":"2023-10-17T01:48:20.041850+0000","mapping_epoch":16,"log_start":"94'430717","ondisk_log_start":"94'430717","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":28,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 14s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":6794399744,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":439634,"num_read_kb":1728424,"num_write":439487,"num_write_kb":1757948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":3703,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:48:20.874464+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T01:48:20.874464+0000","last_peered":"2023-10-17T01:48:20.874464+0000","last_clean":"2023-10-17T01:48:20.874464+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:48:20.874464+0000","last_undegraded":"2023-10-17T01:48:20.874464+0000","last_fullsized":"2023-10-17T01:48:20.874464+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"95'430683","reported_seq":860970,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:48:20.566621+0000","last_change":"2023-10-17T00:42:20.888387+0000","last_active":"2023-10-17T01:48:20.566621+0000","last_peered":"2023-10-17T01:48:20.566621+0000","last_clean":"2023-10-17T01:48:20.566621+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:48:20.566621+0000","last_undegraded":"2023-10-17T01:48:20.566621+0000","last_fullsized":"2023-10-17T01:48:20.566621+0000","mapping_epoch":16,"log_start":"94'420616","ondisk_log_start":"94'420616","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6644932627,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":431414,"num_read_kb":1695632,"num_write":429410,"num_write_kb":1717634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"95'8471","reported_seq":12048,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:48:18.540775+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T01:48:18.540775+0000","last_peered":"2023-10-17T01:48:18.540775+0000","last_clean":"2023-10-17T01:48:18.540775+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:48:18.540775+0000","last_undegraded":"2023-10-17T01:48:18.540775+0000","last_fullsized":"2023-10-17T01:48:18.540775+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":8471,"log_dups_size":0,"ondisk_log_size":8471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":264,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:48:06.751689+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:48:06.751689+0000","last_peered":"2023-10-17T01:48:06.751689+0000","last_clean":"2023-10-17T01:48:06.751689+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:48:06.751689+0000","last_undegraded":"2023-10-17T01:48:06.751689+0000","last_fullsized":"2023-10-17T01:48:06.751689+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26730217491,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1732362,"num_read_kb":6810604,"num_write":1728300,"num_write_kb":6913194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14161682432,"data_stored":17562542118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40286,"ondisk_log_size":40286,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":41401,"ondisk_log_size":41401,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739167,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8709676,"kb_used_data":6917348,"kb_used_omap":560,"kb_used_meta":1791759,"kb_avail":85662164,"statfs":{"total":96636764160,"available":87718055936,"internally_reserved":0,"allocated":7083364352,"data_stored":8783228764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":574073,"internal_metadata":1834761607},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,4,9,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:48:02 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.316,"15min":0.36499999999999999},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":2.2109999999999999,"5min":2.2109999999999999,"15min":4.5030000000000001},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.45400000000000001,"15min":0.53600000000000003},"min":{"1min":0.21099999999999999,"5min":0.20300000000000001,"15min":0.19900000000000001},"max":{"1min":4.2610000000000001,"5min":4.2610000000000001,"15min":5.4240000000000004},"last":0.372}]},{"osd":1,"last update":"Tue Oct 17 01:48:15 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.50900000000000001,"15min":0.51500000000000001},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.191},"max":{"1min":4.3419999999999996,"5min":4.3419999999999996,"15min":12.413},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.57299999999999995,"15min":0.49399999999999999},"min":{"1min":0.249,"5min":0.223,"15min":0.223},"max":{"1min":4.1790000000000003,"5min":5.0529999999999999,"15min":5.282},"last":0.27200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739164,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4691772,"kb_used_data":3405172,"kb_used_omap":284,"kb_used_meta":1286307,"kb_avail":89680068,"statfs":{"total":96636764160,"available":91832389632,"internally_reserved":0,"allocated":3486896128,"data_stored":4324534139,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":291170,"internal_metadata":1317179038},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,1,9,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:47:45 2023","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":0.64100000000000001,"15min":0.45100000000000001},"min":{"1min":0.17799999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":23.177,"5min":23.177,"15min":23.177},"last":0.41399999999999998},{"interface":"front","average":{"1min":1.6579999999999999,"5min":0.68999999999999995,"15min":0.53000000000000003},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.19700000000000001},"max":{"1min":23.207999999999998,"5min":23.207999999999998,"15min":23.207999999999998},"last":0.439}]},{"osd":2,"last update":"Tue Oct 17 01:47:45 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.71299999999999997,"15min":0.57699999999999996},"min":{"1min":0.214,"5min":0.184,"15min":0.182},"max":{"1min":2.7599999999999998,"5min":13.757,"15min":13.757},"last":0.318},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.81299999999999994,"15min":0.65800000000000003},"min":{"1min":0.28199999999999997,"5min":0.24099999999999999,"15min":0.23100000000000001},"max":{"1min":2.8079999999999998,"5min":14.195,"15min":14.195},"last":0.34599999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739164,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4947656,"kb_used_data":3510016,"kb_used_omap":289,"kb_used_meta":1437342,"kb_avail":89424184,"statfs":{"total":96636764160,"available":91570364416,"internally_reserved":0,"allocated":3594256384,"data_stored":4456739906,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":296403,"internal_metadata":1471838765},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,12,2,7,9],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:47:19 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.51500000000000001,"15min":0.55900000000000005},"min":{"1min":0.185,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.86899999999999999,"5min":4.5979999999999999,"15min":15.317},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.48399999999999999,"15min":0.55600000000000005},"min":{"1min":0.216,"5min":0.189,"15min":0.17299999999999999},"max":{"1min":1.2230000000000001,"5min":4.6609999999999996,"15min":7.5789999999999997},"last":0.27200000000000002}]},{"osd":2,"last update":"Tue Oct 17 01:47:19 2023","interfaces":[{"interface":"back","average":{"1min":0.30099999999999999,"5min":0.33600000000000002,"15min":0.38600000000000001},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.57199999999999995,"5min":1.98,"15min":5.7709999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.48899999999999999,"15min":0.57799999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.19400000000000001},"max":{"1min":3.9590000000000001,"5min":3.9590000000000001,"15min":7.6230000000000002},"last":0.309}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3593302016,"data_stored":4455899136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3485818880,"data_stored":4323651603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7082561536,"data_stored":8782991379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:48:25.169 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:48:25.169 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:48:25.170 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:48:25.170 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:48:25.171 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T01:48:30.951 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:48:30.944+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:48:45.172 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:48:45.653 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:48:45.654 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:48:45.693 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2092,"stamp":"2023-10-17T01:48:45.341254+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26764421976,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1763635,"num_read_kb":6860457,"num_write":1737067,"num_write_kb":6947919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81902,"ondisk_log_size":81902,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18420116,"kb_used_data":13894196,"kb_used_omap":1134,"kb_used_meta":4524753,"kb_avail":264695404,"statfs":{"total":289910292480,"available":271048093696,"internally_reserved":0,"allocated":14227656704,"data_stored":17627642649,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1161646,"internal_metadata":4633347666},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,3,8,7,4,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18919424,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4040,"num_read_kb":16136,"num_write":4583,"num_write_kb":18332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002039"},"pg_stats":[{"pgid":"2.7","version":"95'11310","reported_seq":16177,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:48:39.976025+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T01:48:39.976025+0000","last_peered":"2023-10-17T01:48:39.976025+0000","last_clean":"2023-10-17T01:48:39.976025+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:48:39.976025+0000","last_undegraded":"2023-10-17T01:48:39.976025+0000","last_fullsized":"2023-10-17T01:48:39.976025+0000","mapping_epoch":13,"log_start":"50'1300","ondisk_log_start":"50'1300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"95'8521","reported_seq":13341,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:48:39.963031+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T01:48:39.963031+0000","last_peered":"2023-10-17T01:48:39.963031+0000","last_clean":"2023-10-17T01:48:39.963031+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:48:39.963031+0000","last_undegraded":"2023-10-17T01:48:39.963031+0000","last_fullsized":"2023-10-17T01:48:39.963031+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":8521,"log_dups_size":0,"ondisk_log_size":8521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"95'5713","reported_seq":8223,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:48:39.946581+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T01:48:39.946581+0000","last_peered":"2023-10-17T01:48:39.946581+0000","last_clean":"2023-10-17T01:48:39.946581+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:48:39.946581+0000","last_undegraded":"2023-10-17T01:48:39.946581+0000","last_fullsized":"2023-10-17T01:48:39.946581+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":5713,"log_dups_size":0,"ondisk_log_size":5713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"95'8551","reported_seq":12776,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:48:39.963157+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T01:48:39.963157+0000","last_peered":"2023-10-17T01:48:39.963157+0000","last_clean":"2023-10-17T01:48:39.963157+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:48:39.963157+0000","last_undegraded":"2023-10-17T01:48:39.963157+0000","last_fullsized":"2023-10-17T01:48:39.963157+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":8551,"log_dups_size":0,"ondisk_log_size":8551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"95'441471","reported_seq":880807,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:48:45.244213+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T01:48:45.244213+0000","last_peered":"2023-10-17T01:48:45.244213+0000","last_clean":"2023-10-17T01:48:45.244213+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:48:45.244213+0000","last_undegraded":"2023-10-17T01:48:45.244213+0000","last_fullsized":"2023-10-17T01:48:45.244213+0000","mapping_epoch":16,"log_start":"94'431408","ondisk_log_start":"94'431408","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":90,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 32s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":6784143360,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":440473,"num_read_kb":1731068,"num_write":440119,"num_write_kb":1760476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":244,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:48:03.009517+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:48:03.009517+0000","last_peered":"2023-10-17T01:48:03.009517+0000","last_clean":"2023-10-17T01:48:03.009517+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:48:03.009517+0000","last_undegraded":"2023-10-17T01:48:03.009517+0000","last_fullsized":"2023-10-17T01:48:03.009517+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"95'424679","reported_seq":848047,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:48:40.632251+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T01:48:40.632251+0000","last_peered":"2023-10-17T01:48:40.632251+0000","last_clean":"2023-10-17T01:48:40.632251+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:48:40.632251+0000","last_undegraded":"2023-10-17T01:48:40.632251+0000","last_fullsized":"2023-10-17T01:48:40.632251+0000","mapping_epoch":16,"log_start":"94'414616","ondisk_log_start":"94'414616","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":76,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 19s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6523658240,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":424459,"num_read_kb":1669932,"num_write":423387,"num_write_kb":1693548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2471,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:48:41.349014+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T01:48:41.349014+0000","last_peered":"2023-10-17T01:48:41.349014+0000","last_clean":"2023-10-17T01:48:41.349014+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:48:41.349014+0000","last_undegraded":"2023-10-17T01:48:41.349014+0000","last_fullsized":"2023-10-17T01:48:41.349014+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"95'443085","reported_seq":883676,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:48:45.227208+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T01:48:45.227208+0000","last_peered":"2023-10-17T01:48:45.227208+0000","last_clean":"2023-10-17T01:48:45.227208+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:48:45.227208+0000","last_undegraded":"2023-10-17T01:48:45.227208+0000","last_fullsized":"2023-10-17T01:48:45.227208+0000","mapping_epoch":16,"log_start":"94'433017","ondisk_log_start":"94'433017","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":118,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 40s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":6803607552,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":441723,"num_read_kb":1736772,"num_write":441722,"num_write_kb":1766888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":3716,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:48:41.349379+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T01:48:41.349379+0000","last_peered":"2023-10-17T01:48:41.349379+0000","last_clean":"2023-10-17T01:48:41.349379+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:48:41.349379+0000","last_undegraded":"2023-10-17T01:48:41.349379+0000","last_fullsized":"2023-10-17T01:48:41.349379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"95'432482","reported_seq":864385,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:48:40.617536+0000","last_change":"2023-10-17T01:48:39.446076+0000","last_active":"2023-10-17T01:48:40.617536+0000","last_peered":"2023-10-17T01:48:40.617536+0000","last_clean":"2023-10-17T01:48:40.617536+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:48:40.617536+0000","last_undegraded":"2023-10-17T01:48:40.617536+0000","last_fullsized":"2023-10-17T01:48:40.617536+0000","mapping_epoch":16,"log_start":"94'422416","ondisk_log_start":"94'422416","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6652420115,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":433018,"num_read_kb":1702032,"num_write":431209,"num_write_kb":1724830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"95'8520","reported_seq":12109,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:48:39.974545+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T01:48:39.974545+0000","last_peered":"2023-10-17T01:48:39.974545+0000","last_clean":"2023-10-17T01:48:39.974545+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:48:39.974545+0000","last_undegraded":"2023-10-17T01:48:39.974545+0000","last_fullsized":"2023-10-17T01:48:39.974545+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":8520,"log_dups_size":0,"ondisk_log_size":8520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":265,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:48:06.751833+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:48:06.751833+0000","last_peered":"2023-10-17T01:48:06.751833+0000","last_clean":"2023-10-17T01:48:06.751833+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:48:06.751833+0000","last_undegraded":"2023-10-17T01:48:06.751833+0000","last_fullsized":"2023-10-17T01:48:06.751833+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26763829267,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1739673,"num_read_kb":6839804,"num_write":1736437,"num_write_kb":6945742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14225133568,"data_stored":17625993254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40260,"ondisk_log_size":40260,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":41543,"ondisk_log_size":41543,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739171,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8667324,"kb_used_data":6946612,"kb_used_omap":560,"kb_used_meta":1720143,"kb_avail":85704516,"statfs":{"total":96636764160,"available":87761424384,"internally_reserved":0,"allocated":7113330688,"data_stored":8813195100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":574073,"internal_metadata":1761426823},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:48:02 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.316,"15min":0.36499999999999999},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":2.2109999999999999,"5min":2.2109999999999999,"15min":4.5030000000000001},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.45400000000000001,"15min":0.53600000000000003},"min":{"1min":0.21099999999999999,"5min":0.20300000000000001,"15min":0.19900000000000001},"max":{"1min":4.2610000000000001,"5min":4.2610000000000001,"15min":5.4240000000000004},"last":0.27500000000000002}]},{"osd":1,"last update":"Tue Oct 17 01:48:15 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.50900000000000001,"15min":0.51500000000000001},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.191},"max":{"1min":4.3419999999999996,"5min":4.3419999999999996,"15min":12.413},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.57299999999999995,"15min":0.49399999999999999},"min":{"1min":0.249,"5min":0.223,"15min":0.223},"max":{"1min":4.1790000000000003,"5min":5.0529999999999999,"15min":5.282},"last":0.29799999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739168,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4728572,"kb_used_data":3419312,"kb_used_omap":284,"kb_used_meta":1308963,"kb_avail":89643268,"statfs":{"total":96636764160,"available":91794706432,"internally_reserved":0,"allocated":3501375488,"data_stored":4339013499,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":291170,"internal_metadata":1340378782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,2,3,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:47:45 2023","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":0.64100000000000001,"15min":0.45100000000000001},"min":{"1min":0.17799999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":23.177,"5min":23.177,"15min":23.177},"last":0.23899999999999999},{"interface":"front","average":{"1min":1.6579999999999999,"5min":0.68999999999999995,"15min":0.53000000000000003},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.19700000000000001},"max":{"1min":23.207999999999998,"5min":23.207999999999998,"15min":23.207999999999998},"last":0.312}]},{"osd":2,"last update":"Tue Oct 17 01:47:45 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.71299999999999997,"15min":0.57699999999999996},"min":{"1min":0.214,"5min":0.184,"15min":0.182},"max":{"1min":2.7599999999999998,"5min":13.757,"15min":13.757},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.81299999999999994,"15min":0.65800000000000003},"min":{"1min":0.28199999999999997,"5min":0.24099999999999999,"15min":0.23100000000000001},"max":{"1min":2.8079999999999998,"5min":14.195,"15min":14.195},"last":0.36299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739169,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5024220,"kb_used_data":3528272,"kb_used_omap":289,"kb_used_meta":1495646,"kb_avail":89347620,"statfs":{"total":96636764160,"available":91491962880,"internally_reserved":0,"allocated":3612950528,"data_stored":4475434050,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":296403,"internal_metadata":1531542061},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:48:23 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.53800000000000003,"15min":0.57899999999999996},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":5.0709999999999997,"5min":5.0709999999999997,"15min":15.317},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.47399999999999998,"15min":0.54800000000000004},"min":{"1min":0.20100000000000001,"5min":0.189,"15min":0.17299999999999999},"max":{"1min":5.5039999999999996,"5min":5.5039999999999996,"15min":7.5789999999999997},"last":0.215}]},{"osd":2,"last update":"Tue Oct 17 01:48:23 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.32100000000000001,"15min":0.38300000000000001},"min":{"1min":0.20100000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.48599999999999999,"5min":1.98,"15min":5.7709999999999999},"last":0.31},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.48599999999999999,"15min":0.57199999999999995},"min":{"1min":0.218,"5min":0.20599999999999999,"15min":0.19400000000000001},"max":{"1min":3.968,"5min":3.968,"15min":7.6230000000000002},"last":0.252}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3612024832,"data_stored":4474621952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3500421120,"data_stored":4338253843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7112687616,"data_stored":8813117459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:48:45.695 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:48:45.696 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:48:45.696 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:48:45.697 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:48:45.697 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T01:49:05.698 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:49:06.276 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:49:06.277 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:49:06.314 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2103,"stamp":"2023-10-17T01:49:05.344740+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26790366040,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1769649,"num_read_kb":6884433,"num_write":1743370,"num_write_kb":6973131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81939,"ondisk_log_size":81939,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18687512,"kb_used_data":13948096,"kb_used_omap":1134,"kb_used_meta":4738257,"kb_avail":264428008,"statfs":{"total":289910292480,"available":270774280192,"internally_reserved":0,"allocated":14282850304,"data_stored":17682824723,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1161646,"internal_metadata":4851975762},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,4,1,6,6,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15036416,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3489,"num_read_kb":13912,"num_write":3652,"num_write_kb":14608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002150"},"pg_stats":[{"pgid":"2.7","version":"96'11350","reported_seq":16234,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-17T01:48:58.525859+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T01:48:58.525859+0000","last_peered":"2023-10-17T01:48:58.525859+0000","last_clean":"2023-10-17T01:48:58.525859+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:48:58.525859+0000","last_undegraded":"2023-10-17T01:48:58.525859+0000","last_fullsized":"2023-10-17T01:48:58.525859+0000","mapping_epoch":13,"log_start":"50'1300","ondisk_log_start":"50'1300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'8551","reported_seq":13384,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-17T01:48:58.508604+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T01:48:58.508604+0000","last_peered":"2023-10-17T01:48:58.508604+0000","last_clean":"2023-10-17T01:48:58.508604+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:48:58.508604+0000","last_undegraded":"2023-10-17T01:48:58.508604+0000","last_fullsized":"2023-10-17T01:48:58.508604+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":8551,"log_dups_size":0,"ondisk_log_size":8551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'5733","reported_seq":8252,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-17T01:48:58.508482+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T01:48:58.508482+0000","last_peered":"2023-10-17T01:48:58.508482+0000","last_clean":"2023-10-17T01:48:58.508482+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:48:58.508482+0000","last_undegraded":"2023-10-17T01:48:58.508482+0000","last_fullsized":"2023-10-17T01:48:58.508482+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":5733,"log_dups_size":0,"ondisk_log_size":5733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'8581","reported_seq":12819,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-17T01:48:58.508452+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T01:48:58.508452+0000","last_peered":"2023-10-17T01:48:58.508452+0000","last_clean":"2023-10-17T01:48:58.508452+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:48:58.508452+0000","last_undegraded":"2023-10-17T01:48:58.508452+0000","last_fullsized":"2023-10-17T01:48:58.508452+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":8581,"log_dups_size":0,"ondisk_log_size":8581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'442867","reported_seq":883484,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:49:00.317406+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T01:49:00.317406+0000","last_peered":"2023-10-17T01:49:00.317406+0000","last_clean":"2023-10-17T01:49:00.317406+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:49:00.317406+0000","last_undegraded":"2023-10-17T01:49:00.317406+0000","last_fullsized":"2023-10-17T01:49:00.317406+0000","mapping_epoch":16,"log_start":"94'432808","ondisk_log_start":"94'432808","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":140,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 47s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":6789734400,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":441747,"num_read_kb":1736144,"num_write":441488,"num_write_kb":1765952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":245,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:48:03.009614+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:48:03.009614+0000","last_peered":"2023-10-17T01:48:03.009614+0000","last_clean":"2023-10-17T01:48:03.009614+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:48:03.009614+0000","last_undegraded":"2023-10-17T01:48:03.009614+0000","last_fullsized":"2023-10-17T01:48:03.009614+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'426449","reported_seq":851462,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:49:00.748074+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T01:49:00.748074+0000","last_peered":"2023-10-17T01:49:00.748074+0000","last_clean":"2023-10-17T01:49:00.748074+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:49:00.748074+0000","last_undegraded":"2023-10-17T01:49:00.748074+0000","last_fullsized":"2023-10-17T01:49:00.748074+0000","mapping_epoch":16,"log_start":"94'416416","ondisk_log_start":"94'416416","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":146,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 39s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6530740224,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":426089,"num_read_kb":1676424,"num_write":425117,"num_write_kb":1700468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2480,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-17T01:49:01.823483+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T01:49:01.823483+0000","last_peered":"2023-10-17T01:49:01.823483+0000","last_clean":"2023-10-17T01:49:01.823483+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:49:01.823483+0000","last_undegraded":"2023-10-17T01:49:01.823483+0000","last_fullsized":"2023-10-17T01:49:01.823483+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'444528","reported_seq":886472,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:49:00.307970+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T01:49:00.307970+0000","last_peered":"2023-10-17T01:49:00.307970+0000","last_clean":"2023-10-17T01:49:00.307970+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:49:00.307970+0000","last_undegraded":"2023-10-17T01:49:00.307970+0000","last_fullsized":"2023-10-17T01:49:00.307970+0000","mapping_epoch":16,"log_start":"94'434517","ondisk_log_start":"94'434517","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":150,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 55s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":6809456640,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":443072,"num_read_kb":1742164,"num_write":443148,"num_write_kb":1772592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":3729,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-17T01:49:01.823878+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T01:49:01.823878+0000","last_peered":"2023-10-17T01:49:01.823878+0000","last_clean":"2023-10-17T01:49:01.823878+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:49:01.823878+0000","last_undegraded":"2023-10-17T01:49:01.823878+0000","last_fullsized":"2023-10-17T01:49:01.823878+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'434260","reported_seq":867930,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-17T01:49:00.776758+0000","last_change":"2023-10-17T01:49:00.666766+0000","last_active":"2023-10-17T01:49:00.776758+0000","last_peered":"2023-10-17T01:49:00.776758+0000","last_clean":"2023-10-17T01:49:00.776758+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:49:00.776758+0000","last_undegraded":"2023-10-17T01:49:00.776758+0000","last_fullsized":"2023-10-17T01:49:00.776758+0000","mapping_epoch":16,"log_start":"94'424216","ondisk_log_start":"94'424216","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6659842067,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":434779,"num_read_kb":1709048,"num_write":432987,"num_write_kb":1731942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'8550","reported_seq":12152,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-17T01:48:58.542615+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T01:48:58.542615+0000","last_peered":"2023-10-17T01:48:58.542615+0000","last_clean":"2023-10-17T01:48:58.542615+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:48:58.542615+0000","last_undegraded":"2023-10-17T01:48:58.542615+0000","last_fullsized":"2023-10-17T01:48:58.542615+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":8550,"log_dups_size":0,"ondisk_log_size":8550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":266,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:48:35.619155+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:48:35.619155+0000","last_peered":"2023-10-17T01:48:35.619155+0000","last_clean":"2023-10-17T01:48:35.619155+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:48:35.619155+0000","last_undegraded":"2023-10-17T01:48:35.619155+0000","last_fullsized":"2023-10-17T01:48:35.619155+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26789773331,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1745687,"num_read_kb":6863780,"num_write":1742740,"num_write_kb":6970954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14280626176,"data_stored":17681485862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40147,"ondisk_log_size":40147,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":41693,"ondisk_log_size":41693,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739175,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8827536,"kb_used_data":6975240,"kb_used_omap":560,"kb_used_meta":1851727,"kb_avail":85544304,"statfs":{"total":96636764160,"available":87597367296,"internally_reserved":0,"allocated":7142645760,"data_stored":8842506330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":574073,"internal_metadata":1896168839},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:48:02 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.316,"15min":0.36499999999999999},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":2.2109999999999999,"5min":2.2109999999999999,"15min":4.5030000000000001},"last":2.0379999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.45400000000000001,"15min":0.53600000000000003},"min":{"1min":0.21099999999999999,"5min":0.20300000000000001,"15min":0.19900000000000001},"max":{"1min":4.2610000000000001,"5min":4.2610000000000001,"15min":5.4240000000000004},"last":0.29199999999999998}]},{"osd":1,"last update":"Tue Oct 17 01:48:15 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.50900000000000001,"15min":0.51500000000000001},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.191},"max":{"1min":4.3419999999999996,"5min":4.3419999999999996,"15min":12.413},"last":0.309},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.57299999999999995,"15min":0.49399999999999999},"min":{"1min":0.249,"5min":0.223,"15min":0.223},"max":{"1min":4.1790000000000003,"5min":5.0529999999999999,"15min":5.282},"last":0.25800000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739172,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4792648,"kb_used_data":3433408,"kb_used_omap":284,"kb_used_meta":1358947,"kb_avail":89579192,"statfs":{"total":96636764160,"available":91729092608,"internally_reserved":0,"allocated":3515809792,"data_stored":4353443961,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":291170,"internal_metadata":1391562398},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,2,3,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:48:48 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.628,"15min":0.45500000000000002},"min":{"1min":0.218,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.64400000000000002,"5min":23.177,"15min":23.177},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.67700000000000005,"15min":0.53400000000000003},"min":{"1min":0.312,"5min":0.20200000000000001,"15min":0.19700000000000001},"max":{"1min":0.67300000000000004,"5min":23.207999999999998,"15min":23.207999999999998},"last":0.39300000000000002}]},{"osd":2,"last update":"Tue Oct 17 01:48:48 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.627,"15min":0.55900000000000005},"min":{"1min":0.22,"5min":0.184,"15min":0.182},"max":{"1min":0.63700000000000001,"5min":13.757,"15min":13.757},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.72499999999999998,"15min":0.63900000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.23100000000000001},"max":{"1min":0.67100000000000004,"5min":14.195,"15min":14.195},"last":0.61699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739172,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5067328,"kb_used_data":3539448,"kb_used_omap":289,"kb_used_meta":1527582,"kb_avail":89304512,"statfs":{"total":96636764160,"available":91447820288,"internally_reserved":0,"allocated":3624394752,"data_stored":4486874432,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":296403,"internal_metadata":1564244525},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,0,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:48:23 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.53800000000000003,"15min":0.57899999999999996},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":5.0709999999999997,"5min":5.0709999999999997,"15min":15.317},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.47399999999999998,"15min":0.54800000000000004},"min":{"1min":0.20100000000000001,"5min":0.189,"15min":0.17299999999999999},"max":{"1min":5.5039999999999996,"5min":5.5039999999999996,"15min":7.5789999999999997},"last":0.23999999999999999}]},{"osd":2,"last update":"Tue Oct 17 01:48:23 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.32100000000000001,"15min":0.38300000000000001},"min":{"1min":0.20100000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.48599999999999999,"5min":1.98,"15min":5.7709999999999999},"last":0.317},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.48599999999999999,"15min":0.57199999999999995},"min":{"1min":0.218,"5min":0.20599999999999999,"15min":0.19400000000000001},"max":{"1min":3.968,"5min":3.968,"15min":7.6230000000000002},"last":0.26400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3623469056,"data_stored":4486066176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3514802176,"data_stored":4352634899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7142354944,"data_stored":8842784787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:49:06.316 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:49:06.316 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:49:06.317 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:49:06.317 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:49:06.318 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T01:49:26.319 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:49:26.801 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:49:26.803 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:49:26.862 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2113,"stamp":"2023-10-17T01:49:25.350839+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26820033368,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1777037,"num_read_kb":6913893,"num_write":1750621,"num_write_kb":7002135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82138,"ondisk_log_size":82138,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18908780,"kb_used_data":14005844,"kb_used_omap":1134,"kb_used_meta":4901777,"kb_avail":264206740,"statfs":{"total":289910292480,"available":270547701760,"internally_reserved":0,"allocated":14341984256,"data_stored":17741958675,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1161646,"internal_metadata":5019420242},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,4,4,11,8,18,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14811136,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3688,"num_read_kb":14712,"num_write":3627,"num_write_kb":14508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002121"},"pg_stats":[{"pgid":"2.7","version":"96'11390","reported_seq":16290,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-17T01:49:19.320373+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T01:49:19.320373+0000","last_peered":"2023-10-17T01:49:19.320373+0000","last_clean":"2023-10-17T01:49:19.320373+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:49:19.320373+0000","last_undegraded":"2023-10-17T01:49:19.320373+0000","last_fullsized":"2023-10-17T01:49:19.320373+0000","mapping_epoch":13,"log_start":"50'1300","ondisk_log_start":"50'1300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'8581","reported_seq":13426,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-17T01:49:19.323882+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T01:49:19.323882+0000","last_peered":"2023-10-17T01:49:19.323882+0000","last_clean":"2023-10-17T01:49:19.323882+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:49:19.323882+0000","last_undegraded":"2023-10-17T01:49:19.323882+0000","last_fullsized":"2023-10-17T01:49:19.323882+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":8581,"log_dups_size":0,"ondisk_log_size":8581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'5753","reported_seq":8281,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-17T01:49:19.297115+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T01:49:19.297115+0000","last_peered":"2023-10-17T01:49:19.297115+0000","last_clean":"2023-10-17T01:49:19.297115+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:49:19.297115+0000","last_undegraded":"2023-10-17T01:49:19.297115+0000","last_fullsized":"2023-10-17T01:49:19.297115+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":5753,"log_dups_size":0,"ondisk_log_size":5753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'8611","reported_seq":12862,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-17T01:49:19.316145+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T01:49:19.316145+0000","last_peered":"2023-10-17T01:49:19.316145+0000","last_clean":"2023-10-17T01:49:19.316145+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:49:19.316145+0000","last_undegraded":"2023-10-17T01:49:19.316145+0000","last_fullsized":"2023-10-17T01:49:19.316145+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":8611,"log_dups_size":0,"ondisk_log_size":8611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'444732","reported_seq":887226,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:49:20.431697+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T01:49:20.431697+0000","last_peered":"2023-10-17T01:49:20.431697+0000","last_clean":"2023-10-17T01:49:20.431697+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:49:20.431697+0000","last_undegraded":"2023-10-17T01:49:20.431697+0000","last_fullsized":"2023-10-17T01:49:20.431697+0000","mapping_epoch":16,"log_start":"94'434708","ondisk_log_start":"94'434708","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":192,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 67s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":6797230080,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":443615,"num_read_kb":1743600,"num_write":443324,"num_write_kb":1773296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":246,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-17T01:48:52.054370+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:48:52.054370+0000","last_peered":"2023-10-17T01:48:52.054370+0000","last_clean":"2023-10-17T01:48:52.054370+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:48:52.054370+0000","last_undegraded":"2023-10-17T01:48:52.054370+0000","last_fullsized":"2023-10-17T01:48:52.054370+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'428260","reported_seq":855096,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:49:20.901198+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T01:49:20.901198+0000","last_peered":"2023-10-17T01:49:20.901198+0000","last_clean":"2023-10-17T01:49:20.901198+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:49:20.901198+0000","last_undegraded":"2023-10-17T01:49:20.901198+0000","last_fullsized":"2023-10-17T01:49:20.901198+0000","mapping_epoch":16,"log_start":"94'418216","ondisk_log_start":"94'418216","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":198,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 59s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6538031104,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":427901,"num_read_kb":1683648,"num_write":426897,"num_write_kb":1707588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2489,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-17T01:49:22.310248+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T01:49:22.310248+0000","last_peered":"2023-10-17T01:49:22.310248+0000","last_clean":"2023-10-17T01:49:22.310248+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:49:22.310248+0000","last_undegraded":"2023-10-17T01:49:22.310248+0000","last_fullsized":"2023-10-17T01:49:22.310248+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'446375","reported_seq":890193,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:49:20.425090+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T01:49:20.425090+0000","last_peered":"2023-10-17T01:49:20.425090+0000","last_clean":"2023-10-17T01:49:20.425090+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:49:20.425090+0000","last_undegraded":"2023-10-17T01:49:20.425090+0000","last_fullsized":"2023-10-17T01:49:20.425090+0000","mapping_epoch":16,"log_start":"94'436317","ondisk_log_start":"94'436317","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":214,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 75s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":6816854016,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":444936,"num_read_kb":1749604,"num_write":444957,"num_write_kb":1779828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":3741,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-17T01:49:22.369844+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T01:49:22.369844+0000","last_peered":"2023-10-17T01:49:22.369844+0000","last_clean":"2023-10-17T01:49:22.369844+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:49:22.369844+0000","last_undegraded":"2023-10-17T01:49:22.369844+0000","last_fullsized":"2023-10-17T01:49:22.369844+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'436086","reported_seq":871605,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-17T01:49:20.893058+0000","last_change":"2023-10-17T01:49:20.227673+0000","last_active":"2023-10-17T01:49:20.893058+0000","last_peered":"2023-10-17T01:49:20.893058+0000","last_clean":"2023-10-17T01:49:20.893058+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:49:20.893058+0000","last_undegraded":"2023-10-17T01:49:20.893058+0000","last_fullsized":"2023-10-17T01:49:20.893058+0000","mapping_epoch":16,"log_start":"94'426016","ondisk_log_start":"94'426016","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6667325459,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":436623,"num_read_kb":1716388,"num_write":434813,"num_write_kb":1739246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'8580","reported_seq":12195,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-17T01:49:19.344465+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T01:49:19.344465+0000","last_peered":"2023-10-17T01:49:19.344465+0000","last_clean":"2023-10-17T01:49:19.344465+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:49:19.344465+0000","last_undegraded":"2023-10-17T01:49:19.344465+0000","last_fullsized":"2023-10-17T01:49:19.344465+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":8580,"log_dups_size":0,"ondisk_log_size":8580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":266,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:48:35.619155+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:48:35.619155+0000","last_peered":"2023-10-17T01:48:35.619155+0000","last_clean":"2023-10-17T01:48:35.619155+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:48:35.619155+0000","last_undegraded":"2023-10-17T01:48:35.619155+0000","last_fullsized":"2023-10-17T01:48:35.619155+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26819440659,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1753075,"num_read_kb":6893240,"num_write":1749991,"num_write_kb":6999958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14340063232,"data_stored":17740922918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40196,"ondisk_log_size":40196,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":41843,"ondisk_log_size":41843,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739179,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8938160,"kb_used_data":7003944,"kb_used_omap":560,"kb_used_meta":1933647,"kb_avail":85433680,"statfs":{"total":96636764160,"available":87484088320,"internally_reserved":0,"allocated":7172038656,"data_stored":8871899226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":574073,"internal_metadata":1980054919},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,3,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:49:03 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.34000000000000002,"15min":0.36799999999999999},"min":{"1min":0.189,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":2.0379999999999998,"5min":2.2109999999999999,"15min":4.5030000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.222,"5min":0.20300000000000001,"15min":0.19900000000000001},"max":{"1min":0.70599999999999996,"5min":4.2610000000000001,"15min":5.4240000000000004},"last":0.82199999999999995}]},{"osd":1,"last update":"Tue Oct 17 01:49:16 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.45000000000000001,"15min":0.51400000000000001},"min":{"1min":0.23300000000000001,"5min":0.19400000000000001,"15min":0.191},"max":{"1min":1.194,"5min":4.3419999999999996,"15min":12.413},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.45300000000000001,"15min":0.48899999999999999},"min":{"1min":0.25800000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.56699999999999995,"5min":4.1790000000000003,"15min":5.282},"last":0.56899999999999995}]}]},{"osd":1,"up_from":8,"seq":34359739176,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4825104,"kb_used_data":3447816,"kb_used_omap":284,"kb_used_meta":1376995,"kb_avail":89546736,"statfs":{"total":96636764160,"available":91695857664,"internally_reserved":0,"allocated":3530563584,"data_stored":4368197753,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":291170,"internal_metadata":1410043550},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,1,3,2,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:48:48 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.628,"15min":0.45500000000000002},"min":{"1min":0.218,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.64400000000000002,"5min":23.177,"15min":23.177},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.67700000000000005,"15min":0.53400000000000003},"min":{"1min":0.312,"5min":0.20200000000000001,"15min":0.19700000000000001},"max":{"1min":0.67300000000000004,"5min":23.207999999999998,"15min":23.207999999999998},"last":0.27300000000000002}]},{"osd":2,"last update":"Tue Oct 17 01:48:48 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.627,"15min":0.55900000000000005},"min":{"1min":0.22,"5min":0.184,"15min":0.182},"max":{"1min":0.63700000000000001,"5min":13.757,"15min":13.757},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.72499999999999998,"15min":0.63900000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.23100000000000001},"max":{"1min":0.67100000000000004,"5min":14.195,"15min":14.195},"last":0.47599999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739176,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5145516,"kb_used_data":3554084,"kb_used_omap":289,"kb_used_meta":1591134,"kb_avail":89226324,"statfs":{"total":96636764160,"available":91367755776,"internally_reserved":0,"allocated":3639382016,"data_stored":4501861696,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":296403,"internal_metadata":1629321773},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,5,2,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:48:23 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.53800000000000003,"15min":0.57899999999999996},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":5.0709999999999997,"5min":5.0709999999999997,"15min":15.317},"last":0.374},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.47399999999999998,"15min":0.54800000000000004},"min":{"1min":0.20100000000000001,"5min":0.189,"15min":0.17299999999999999},"max":{"1min":5.5039999999999996,"5min":5.5039999999999996,"15min":7.5789999999999997},"last":0.53100000000000003}]},{"osd":2,"last update":"Tue Oct 17 01:48:23 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.32100000000000001,"15min":0.38300000000000001},"min":{"1min":0.20100000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.48599999999999999,"5min":1.98,"15min":5.7709999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.48599999999999999,"15min":0.57199999999999995},"min":{"1min":0.218,"5min":0.20599999999999999,"15min":0.19400000000000001},"max":{"1min":3.968,"5min":3.968,"15min":7.6230000000000002},"last":0.22900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3638403072,"data_stored":4501000192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3529560064,"data_stored":4367392787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7172100096,"data_stored":8872529939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:49:26.863 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:49:26.864 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:49:26.864 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:49:26.865 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:49:26.865 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T01:49:46.866 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:49:47.477 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:49:47.479 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:49:47.519 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2124,"stamp":"2023-10-17T01:49:47.357084+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26857794392,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1786386,"num_read_kb":6951113,"num_write":1759776,"num_write_kb":7038755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82288,"ondisk_log_size":82288,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19213108,"kb_used_data":14071380,"kb_used_omap":1134,"kb_used_meta":5140561,"kb_avail":263902412,"statfs":{"total":289910292480,"available":270236069888,"internally_reserved":0,"allocated":14409093120,"data_stored":17809067539,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1161646,"internal_metadata":5263935058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,8,17,8,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22474752,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5670,"num_read_kb":22556,"num_write":5476,"num_write_kb":21904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002005"},"pg_stats":[{"pgid":"2.7","version":"96'11430","reported_seq":16351,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-17T01:49:44.601837+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T01:49:44.601837+0000","last_peered":"2023-10-17T01:49:44.601837+0000","last_clean":"2023-10-17T01:49:44.601837+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:49:44.601837+0000","last_undegraded":"2023-10-17T01:49:44.601837+0000","last_fullsized":"2023-10-17T01:49:44.601837+0000","mapping_epoch":13,"log_start":"52'1400","ondisk_log_start":"52'1400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'8611","reported_seq":13472,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-17T01:49:44.582804+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T01:49:44.582804+0000","last_peered":"2023-10-17T01:49:44.582804+0000","last_clean":"2023-10-17T01:49:44.582804+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:49:44.582804+0000","last_undegraded":"2023-10-17T01:49:44.582804+0000","last_fullsized":"2023-10-17T01:49:44.582804+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":8611,"log_dups_size":0,"ondisk_log_size":8611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'5773","reported_seq":8311,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-17T01:49:44.562036+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T01:49:44.562036+0000","last_peered":"2023-10-17T01:49:44.562036+0000","last_clean":"2023-10-17T01:49:44.562036+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:49:44.562036+0000","last_undegraded":"2023-10-17T01:49:44.562036+0000","last_fullsized":"2023-10-17T01:49:44.562036+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":5773,"log_dups_size":0,"ondisk_log_size":5773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'8641","reported_seq":12907,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-17T01:49:44.582845+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T01:49:44.582845+0000","last_peered":"2023-10-17T01:49:44.582845+0000","last_clean":"2023-10-17T01:49:44.582845+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:49:44.582845+0000","last_undegraded":"2023-10-17T01:49:44.582845+0000","last_fullsized":"2023-10-17T01:49:44.582845+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":8641,"log_dups_size":0,"ondisk_log_size":8641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'447103","reported_seq":891968,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:49:45.614554+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T01:49:45.614554+0000","last_peered":"2023-10-17T01:49:45.614554+0000","last_clean":"2023-10-17T01:49:45.614554+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:49:45.614554+0000","last_undegraded":"2023-10-17T01:49:45.614554+0000","last_fullsized":"2023-10-17T01:49:45.614554+0000","mapping_epoch":16,"log_start":"95'437008","ondisk_log_start":"95'437008","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":278,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 92s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":6806867968,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":445977,"num_read_kb":1753012,"num_write":445648,"num_write_kb":1782592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":246,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-17T01:48:52.054370+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:48:52.054370+0000","last_peered":"2023-10-17T01:48:52.054370+0000","last_clean":"2023-10-17T01:48:52.054370+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:48:52.054370+0000","last_undegraded":"2023-10-17T01:48:52.054370+0000","last_fullsized":"2023-10-17T01:48:52.054370+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'430508","reported_seq":859648,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:49:46.054546+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T01:49:46.054546+0000","last_peered":"2023-10-17T01:49:46.054546+0000","last_clean":"2023-10-17T01:49:46.054546+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:49:46.054546+0000","last_undegraded":"2023-10-17T01:49:46.054546+0000","last_fullsized":"2023-10-17T01:49:46.054546+0000","mapping_epoch":16,"log_start":"95'420417","ondisk_log_start":"95'420417","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":298,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 84s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6547070976,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":430196,"num_read_kb":1692788,"num_write":429089,"num_write_kb":1716356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2497,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-17T01:49:42.803454+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T01:49:42.803454+0000","last_peered":"2023-10-17T01:49:42.803454+0000","last_clean":"2023-10-17T01:49:42.803454+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:49:42.803454+0000","last_undegraded":"2023-10-17T01:49:42.803454+0000","last_fullsized":"2023-10-17T01:49:42.803454+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'448730","reported_seq":894988,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:49:45.578560+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T01:49:45.578560+0000","last_peered":"2023-10-17T01:49:45.578560+0000","last_clean":"2023-10-17T01:49:45.578560+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:49:45.578560+0000","last_undegraded":"2023-10-17T01:49:45.578560+0000","last_fullsized":"2023-10-17T01:49:45.578560+0000","mapping_epoch":16,"log_start":"95'438717","ondisk_log_start":"95'438717","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":292,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 100s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":6826369024,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":447365,"num_read_kb":1759280,"num_write":447269,"num_write_kb":1789076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":3753,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-17T01:49:42.790481+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T01:49:42.790481+0000","last_peered":"2023-10-17T01:49:42.790481+0000","last_clean":"2023-10-17T01:49:42.790481+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:49:42.790481+0000","last_undegraded":"2023-10-17T01:49:42.790481+0000","last_fullsized":"2023-10-17T01:49:42.790481+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'438413","reported_seq":876198,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-17T01:49:46.062621+0000","last_change":"2023-10-17T01:49:43.699474+0000","last_active":"2023-10-17T01:49:46.062621+0000","last_peered":"2023-10-17T01:49:46.062621+0000","last_clean":"2023-10-17T01:49:46.062621+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:49:46.062621+0000","last_undegraded":"2023-10-17T01:49:46.062621+0000","last_fullsized":"2023-10-17T01:49:46.062621+0000","mapping_epoch":16,"log_start":"95'428316","ondisk_log_start":"95'428316","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6676893715,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":438886,"num_read_kb":1725380,"num_write":437140,"num_write_kb":1748554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'8610","reported_seq":12240,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-17T01:49:44.620050+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T01:49:44.620050+0000","last_peered":"2023-10-17T01:49:44.620050+0000","last_clean":"2023-10-17T01:49:44.620050+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:49:44.620050+0000","last_undegraded":"2023-10-17T01:49:44.620050+0000","last_fullsized":"2023-10-17T01:49:44.620050+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":8610,"log_dups_size":0,"ondisk_log_size":8610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":266,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-17T01:48:35.619155+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:48:35.619155+0000","last_peered":"2023-10-17T01:48:35.619155+0000","last_clean":"2023-10-17T01:48:35.619155+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:48:35.619155+0000","last_undegraded":"2023-10-17T01:48:35.619155+0000","last_fullsized":"2023-10-17T01:48:35.619155+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26857201683,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1762424,"num_read_kb":6930460,"num_write":1759146,"num_write_kb":7036578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14407512064,"data_stored":17808371750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40296,"ondisk_log_size":40296,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":41893,"ondisk_log_size":41893,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739183,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9053384,"kb_used_data":7033080,"kb_used_omap":560,"kb_used_meta":2019727,"kb_avail":85318456,"statfs":{"total":96636764160,"available":87366098944,"internally_reserved":0,"allocated":7201873920,"data_stored":8901734490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":574073,"internal_metadata":2068200839},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:49:03 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.34000000000000002,"15min":0.36799999999999999},"min":{"1min":0.189,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":2.0379999999999998,"5min":2.2109999999999999,"15min":4.5030000000000001},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.222,"5min":0.20300000000000001,"15min":0.19900000000000001},"max":{"1min":0.70599999999999996,"5min":4.2610000000000001,"15min":5.4240000000000004},"last":0.29799999999999999}]},{"osd":1,"last update":"Tue Oct 17 01:49:16 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.45000000000000001,"15min":0.51400000000000001},"min":{"1min":0.23300000000000001,"5min":0.19400000000000001,"15min":0.191},"max":{"1min":1.194,"5min":4.3419999999999996,"15min":12.413},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.45300000000000001,"15min":0.48899999999999999},"min":{"1min":0.25800000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.56699999999999995,"5min":4.1790000000000003,"15min":5.282},"last":0.19500000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739181,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4919368,"kb_used_data":3465728,"kb_used_omap":284,"kb_used_meta":1453347,"kb_avail":89452472,"statfs":{"total":96636764160,"available":91599331328,"internally_reserved":0,"allocated":3548905472,"data_stored":4386539641,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":291170,"internal_metadata":1488227998},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:48:48 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.628,"15min":0.45500000000000002},"min":{"1min":0.218,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.64400000000000002,"5min":23.177,"15min":23.177},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.67700000000000005,"15min":0.53400000000000003},"min":{"1min":0.312,"5min":0.20200000000000001,"15min":0.19700000000000001},"max":{"1min":0.67300000000000004,"5min":23.207999999999998,"15min":23.207999999999998},"last":0.32000000000000001}]},{"osd":2,"last update":"Tue Oct 17 01:48:48 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.627,"15min":0.55900000000000005},"min":{"1min":0.22,"5min":0.184,"15min":0.182},"max":{"1min":0.63700000000000001,"5min":13.757,"15min":13.757},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.72499999999999998,"15min":0.63900000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.23100000000000001},"max":{"1min":0.67100000000000004,"5min":14.195,"15min":14.195},"last":0.34499999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739181,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5240356,"kb_used_data":3572572,"kb_used_omap":289,"kb_used_meta":1667486,"kb_avail":89131484,"statfs":{"total":96636764160,"available":91270639616,"internally_reserved":0,"allocated":3658313728,"data_stored":4520793408,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":296403,"internal_metadata":1707506221},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,14,5,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:49:24 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.59799999999999998,"15min":0.60599999999999998},"min":{"1min":0.18099999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":5.9829999999999997,"5min":5.9829999999999997,"15min":15.317},"last":0.75},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.505,"15min":0.56200000000000006},"min":{"1min":0.215,"5min":0.189,"15min":0.184},"max":{"1min":6.0750000000000002,"5min":6.0750000000000002,"15min":7.5789999999999997},"last":0.93200000000000005}]},{"osd":2,"last update":"Tue Oct 17 01:49:24 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.33600000000000002,"15min":0.38200000000000001},"min":{"1min":0.22,"5min":0.20100000000000001,"15min":0.18099999999999999},"max":{"1min":2.2000000000000002,"5min":2.2000000000000002,"15min":5.7709999999999999},"last":0.31},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.48599999999999999,"15min":0.57799999999999996},"min":{"1min":0.21099999999999999,"5min":0.20599999999999999,"15min":0.19400000000000001},"max":{"1min":6.1269999999999998,"5min":6.1269999999999998,"15min":7.6230000000000002},"last":0.29099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3657396224,"data_stored":4519993344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3548086272,"data_stored":4385918995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7202029568,"data_stored":8902459411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:49:47.521 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:49:47.522 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:49:47.522 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:49:47.522 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:49:47.523 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T01:49:51.356 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:49:51.342+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:49:51.361 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:49:51.350+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:49:51.369 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:49:51.362+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:50:04.618 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:50:04.614+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:50:04.619 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:50:04.614+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:50:04.628 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:50:04.626+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:50:04.644 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:50:04.642+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:50:05.028 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:50:05.022+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:50:05.044 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:50:05.038+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:50:05.049 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:50:05.046+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:50:05.050 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:50:05.046+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:50:05.050 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:50:05.046+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:50:05.050 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:50:05.046+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:50:05.051 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:50:05.046+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:50:05.051 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:50:05.046+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:50:05.051 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:50:05.046+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:50:05.051 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:50:05.046+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:50:05.052 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:50:05.046+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:50:05.052 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:50:05.046+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:50:05.052 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:50:05.046+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:50:07.523 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:50:08.099 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:50:08.100 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:50:08.131 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2135,"stamp":"2023-10-17T01:50:07.360458+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26887285592,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1793625,"num_read_kb":6979909,"num_write":1767046,"num_write_kb":7067835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82331,"ondisk_log_size":82331,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19484652,"kb_used_data":14129492,"kb_used_omap":1134,"kb_used_meta":5354001,"kb_avail":263630868,"statfs":{"total":289910292480,"available":269958008832,"internally_reserved":0,"allocated":14468599808,"data_stored":17868562572,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1161646,"internal_metadata":5482497618},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,1,8,11,15,17],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14606336,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3443,"num_read_kb":13684,"num_write":3600,"num_write_kb":14400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.281475"},"pg_stats":[{"pgid":"2.7","version":"97'11504","reported_seq":16442,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:50:05.073827+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T01:50:05.073827+0000","last_peered":"2023-10-17T01:50:05.073827+0000","last_clean":"2023-10-17T01:50:05.073827+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:50:05.073827+0000","last_undegraded":"2023-10-17T01:50:05.073827+0000","last_fullsized":"2023-10-17T01:50:05.073827+0000","mapping_epoch":13,"log_start":"52'1500","ondisk_log_start":"52'1500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"97'8666","reported_seq":13540,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:50:05.058790+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T01:50:05.058790+0000","last_peered":"2023-10-17T01:50:05.058790+0000","last_clean":"2023-10-17T01:50:05.058790+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:50:05.058790+0000","last_undegraded":"2023-10-17T01:50:05.058790+0000","last_fullsized":"2023-10-17T01:50:05.058790+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":8666,"log_dups_size":0,"ondisk_log_size":8666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"97'5811","reported_seq":8358,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:50:05.038959+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T01:50:05.038959+0000","last_peered":"2023-10-17T01:50:05.038959+0000","last_clean":"2023-10-17T01:50:05.038959+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:50:05.038959+0000","last_undegraded":"2023-10-17T01:50:05.038959+0000","last_fullsized":"2023-10-17T01:50:05.038959+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":5811,"log_dups_size":0,"ondisk_log_size":5811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"97'8697","reported_seq":12976,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:50:05.059010+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T01:50:05.059010+0000","last_peered":"2023-10-17T01:50:05.059010+0000","last_clean":"2023-10-17T01:50:05.059010+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:50:05.059010+0000","last_undegraded":"2023-10-17T01:50:05.059010+0000","last_fullsized":"2023-10-17T01:50:05.059010+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":8697,"log_dups_size":0,"ondisk_log_size":8697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"97'448951","reported_seq":895640,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:50:05.798417+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T01:50:05.798417+0000","last_peered":"2023-10-17T01:50:05.798417+0000","last_clean":"2023-10-17T01:50:05.798417+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:50:05.798417+0000","last_undegraded":"2023-10-17T01:50:05.798417+0000","last_fullsized":"2023-10-17T01:50:05.798417+0000","mapping_epoch":16,"log_start":"95'438908","ondisk_log_start":"95'438908","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":328,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 112s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":6814236672,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":447785,"num_read_kb":1760176,"num_write":447465,"num_write_kb":1789860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":247,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-17T01:49:15.415707+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:49:15.415707+0000","last_peered":"2023-10-17T01:49:15.415707+0000","last_clean":"2023-10-17T01:49:15.415707+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:49:15.415707+0000","last_undegraded":"2023-10-17T01:49:15.415707+0000","last_fullsized":"2023-10-17T01:49:15.415707+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"97'432332","reported_seq":863239,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:50:06.260159+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T01:50:06.260159+0000","last_peered":"2023-10-17T01:50:06.260159+0000","last_clean":"2023-10-17T01:50:06.260159+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:50:06.260159+0000","last_undegraded":"2023-10-17T01:50:06.260159+0000","last_fullsized":"2023-10-17T01:50:06.260159+0000","mapping_epoch":16,"log_start":"95'422317","ondisk_log_start":"95'422317","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":350,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 104s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6554361856,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":431949,"num_read_kb":1699768,"num_write":430886,"num_write_kb":1723544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2506,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:50:03.324827+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T01:50:03.324827+0000","last_peered":"2023-10-17T01:50:03.324827+0000","last_clean":"2023-10-17T01:50:03.324827+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:50:03.324827+0000","last_undegraded":"2023-10-17T01:50:03.324827+0000","last_fullsized":"2023-10-17T01:50:03.324827+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"97'450615","reported_seq":898760,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:50:05.770984+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T01:50:05.770984+0000","last_peered":"2023-10-17T01:50:05.770984+0000","last_clean":"2023-10-17T01:50:05.770984+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:50:05.770984+0000","last_undegraded":"2023-10-17T01:50:05.770984+0000","last_fullsized":"2023-10-17T01:50:05.770984+0000","mapping_epoch":16,"log_start":"95'440517","ondisk_log_start":"95'440517","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":362,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 120s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":6833868800,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":449243,"num_read_kb":1766772,"num_write":449118,"num_write_kb":1796472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":3766,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:50:03.315694+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T01:50:03.315694+0000","last_peered":"2023-10-17T01:50:03.315694+0000","last_clean":"2023-10-17T01:50:03.315694+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:50:03.315694+0000","last_undegraded":"2023-10-17T01:50:03.315694+0000","last_fullsized":"2023-10-17T01:50:03.315694+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"97'440220","reported_seq":879812,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:50:06.262939+0000","last_change":"2023-10-17T01:50:03.709409+0000","last_active":"2023-10-17T01:50:06.262939+0000","last_peered":"2023-10-17T01:50:06.262939+0000","last_clean":"2023-10-17T01:50:06.262939+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:50:06.262939+0000","last_undegraded":"2023-10-17T01:50:06.262939+0000","last_fullsized":"2023-10-17T01:50:06.262939+0000","mapping_epoch":16,"log_start":"95'430216","ondisk_log_start":"95'430216","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6684225555,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":440686,"num_read_kb":1732540,"num_write":438947,"num_write_kb":1755782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"97'8666","reported_seq":12309,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:50:05.118362+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T01:50:05.118362+0000","last_peered":"2023-10-17T01:50:05.118362+0000","last_clean":"2023-10-17T01:50:05.118362+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:50:05.118362+0000","last_undegraded":"2023-10-17T01:50:05.118362+0000","last_fullsized":"2023-10-17T01:50:05.118362+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":8666,"log_dups_size":0,"ondisk_log_size":8666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":267,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-17T01:48:52.052685+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:48:52.052685+0000","last_peered":"2023-10-17T01:48:52.052685+0000","last_clean":"2023-10-17T01:48:52.052685+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:48:52.052685+0000","last_undegraded":"2023-10-17T01:48:52.052685+0000","last_fullsized":"2023-10-17T01:48:52.052685+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26886692883,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1769663,"num_read_kb":6959256,"num_write":1766416,"num_write_kb":7065658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14467432448,"data_stored":17868292134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40160,"ondisk_log_size":40160,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":42072,"ondisk_log_size":42072,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739187,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9203252,"kb_used_data":7062188,"kb_used_omap":560,"kb_used_meta":2140495,"kb_avail":85168588,"statfs":{"total":96636764160,"available":87212634112,"internally_reserved":0,"allocated":7231680512,"data_stored":8931537197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":574073,"internal_metadata":2191867271},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:49:03 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.34000000000000002,"15min":0.36799999999999999},"min":{"1min":0.189,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":2.0379999999999998,"5min":2.2109999999999999,"15min":4.5030000000000001},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.222,"5min":0.20300000000000001,"15min":0.19900000000000001},"max":{"1min":0.70599999999999996,"5min":4.2610000000000001,"15min":5.4240000000000004},"last":0.33800000000000002}]},{"osd":1,"last update":"Tue Oct 17 01:49:16 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.45000000000000001,"15min":0.51400000000000001},"min":{"1min":0.23300000000000001,"5min":0.19400000000000001,"15min":0.191},"max":{"1min":1.194,"5min":4.3419999999999996,"15min":12.413},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.45300000000000001,"15min":0.48899999999999999},"min":{"1min":0.25800000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.56699999999999995,"5min":4.1790000000000003,"15min":5.282},"last":0.318}]}]},{"osd":1,"up_from":8,"seq":34359739185,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4990264,"kb_used_data":3480048,"kb_used_omap":284,"kb_used_meta":1509923,"kb_avail":89381576,"statfs":{"total":96636764160,"available":91526733824,"internally_reserved":0,"allocated":3563569152,"data_stored":4401199436,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":291170,"internal_metadata":1546161822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,1,1,4,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:49:49 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.64500000000000002,"15min":0.45400000000000001},"min":{"1min":0.17100000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":2.0089999999999999,"5min":23.177,"15min":23.177},"last":0.254},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.69599999999999995,"15min":0.53400000000000003},"min":{"1min":0.23100000000000001,"5min":0.216,"15min":0.19700000000000001},"max":{"1min":2.0419999999999998,"5min":23.207999999999998,"15min":23.207999999999998},"last":0.27500000000000002}]},{"osd":2,"last update":"Tue Oct 17 01:49:49 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.41499999999999998,"15min":0.52400000000000002},"min":{"1min":0.21199999999999999,"5min":0.184,"15min":0.182},"max":{"1min":0.57999999999999996,"5min":5.1379999999999999,"15min":13.757},"last":0.188},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.46100000000000002,"15min":0.60299999999999998},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.23100000000000001},"max":{"1min":0.61699999999999999,"5min":5.1799999999999997,"15min":14.195},"last":0.24199999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739185,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5291136,"kb_used_data":3587256,"kb_used_omap":289,"kb_used_meta":1703582,"kb_avail":89080704,"statfs":{"total":96636764160,"available":91218640896,"internally_reserved":0,"allocated":3673350144,"data_stored":4535825939,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":296403,"internal_metadata":1744468525},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,9,9,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:49:24 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.59799999999999998,"15min":0.60599999999999998},"min":{"1min":0.18099999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":5.9829999999999997,"5min":5.9829999999999997,"15min":15.317},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.505,"15min":0.56200000000000006},"min":{"1min":0.215,"5min":0.189,"15min":0.184},"max":{"1min":6.0750000000000002,"5min":6.0750000000000002,"15min":7.5789999999999997},"last":0.79200000000000004}]},{"osd":2,"last update":"Tue Oct 17 01:49:24 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.33600000000000002,"15min":0.38200000000000001},"min":{"1min":0.22,"5min":0.20100000000000001,"15min":0.18099999999999999},"max":{"1min":2.2000000000000002,"5min":2.2000000000000002,"15min":5.7709999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.48599999999999999,"15min":0.57799999999999996},"min":{"1min":0.21099999999999999,"5min":0.20599999999999999,"15min":0.19400000000000001},"max":{"1min":6.1269999999999998,"5min":6.1269999999999998,"15min":7.6230000000000002},"last":0.53900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3672416256,"data_stored":4535013376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3562835968,"data_stored":4400668691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7232180224,"data_stored":8932610067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:50:08.132 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:50:08.133 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:50:08.133 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:50:08.134 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:50:08.134 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T01:50:28.135 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:50:28.726 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:50:28.727 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:50:28.774 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2145,"stamp":"2023-10-17T01:50:27.364796+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26916092760,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1800783,"num_read_kb":7008381,"num_write":1774147,"num_write_kb":7096239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82570,"ondisk_log_size":82570,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18874728,"kb_used_data":14186768,"kb_used_omap":1136,"kb_used_meta":4686799,"kb_avail":264240792,"statfs":{"total":289910292480,"available":270582571008,"internally_reserved":0,"allocated":14527250432,"data_stored":17927213196,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1163326,"internal_metadata":4799283138},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,9,8,8,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21692416,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5376,"num_read_kb":21388,"num_write":5345,"num_write_kb":21380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002042"},"pg_stats":[{"pgid":"2.7","version":"97'11536","reported_seq":16490,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:50:24.413011+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T01:50:24.413011+0000","last_peered":"2023-10-17T01:50:24.413011+0000","last_clean":"2023-10-17T01:50:24.413011+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:50:24.413011+0000","last_undegraded":"2023-10-17T01:50:24.413011+0000","last_fullsized":"2023-10-17T01:50:24.413011+0000","mapping_epoch":13,"log_start":"52'1500","ondisk_log_start":"52'1500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"97'8690","reported_seq":13576,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:50:24.392007+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T01:50:24.392007+0000","last_peered":"2023-10-17T01:50:24.392007+0000","last_clean":"2023-10-17T01:50:24.392007+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:50:24.392007+0000","last_undegraded":"2023-10-17T01:50:24.392007+0000","last_fullsized":"2023-10-17T01:50:24.392007+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":8690,"log_dups_size":0,"ondisk_log_size":8690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"97'5827","reported_seq":8382,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:50:24.377668+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T01:50:24.377668+0000","last_peered":"2023-10-17T01:50:24.377668+0000","last_clean":"2023-10-17T01:50:24.377668+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:50:24.377668+0000","last_undegraded":"2023-10-17T01:50:24.377668+0000","last_fullsized":"2023-10-17T01:50:24.377668+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":5827,"log_dups_size":0,"ondisk_log_size":5827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"97'8721","reported_seq":13012,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:50:24.395490+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T01:50:24.395490+0000","last_peered":"2023-10-17T01:50:24.395490+0000","last_clean":"2023-10-17T01:50:24.395490+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:50:24.395490+0000","last_undegraded":"2023-10-17T01:50:24.395490+0000","last_fullsized":"2023-10-17T01:50:24.395490+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":8721,"log_dups_size":0,"ondisk_log_size":8721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"97'450794","reported_seq":899258,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:50:25.940890+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T01:50:25.940890+0000","last_peered":"2023-10-17T01:50:25.940890+0000","last_clean":"2023-10-17T01:50:25.940890+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:50:25.940890+0000","last_undegraded":"2023-10-17T01:50:25.940890+0000","last_fullsized":"2023-10-17T01:50:25.940890+0000","mapping_epoch":16,"log_start":"95'440708","ondisk_log_start":"95'440708","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":396,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 132s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":6821564416,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":449558,"num_read_kb":1767212,"num_write":449271,"num_write_kb":1797084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":247,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-17T01:49:15.415707+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:49:15.415707+0000","last_peered":"2023-10-17T01:49:15.415707+0000","last_clean":"2023-10-17T01:49:15.415707+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:49:15.415707+0000","last_undegraded":"2023-10-17T01:49:15.415707+0000","last_fullsized":"2023-10-17T01:49:15.415707+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"97'434095","reported_seq":866781,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:50:26.392221+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T01:50:26.392221+0000","last_peered":"2023-10-17T01:50:26.392221+0000","last_clean":"2023-10-17T01:50:26.392221+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:50:26.392221+0000","last_undegraded":"2023-10-17T01:50:26.392221+0000","last_fullsized":"2023-10-17T01:50:26.392221+0000","mapping_epoch":16,"log_start":"95'424017","ondisk_log_start":"95'424017","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":412,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 124s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6561374208,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":433721,"num_read_kb":1706828,"num_write":432612,"num_write_kb":1730448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2514,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:50:23.743600+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T01:50:23.743600+0000","last_peered":"2023-10-17T01:50:23.743600+0000","last_clean":"2023-10-17T01:50:23.743600+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:50:23.743600+0000","last_undegraded":"2023-10-17T01:50:23.743600+0000","last_fullsized":"2023-10-17T01:50:23.743600+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"97'452455","reported_seq":902435,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:50:25.940237+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T01:50:25.940237+0000","last_peered":"2023-10-17T01:50:25.940237+0000","last_clean":"2023-10-17T01:50:25.940237+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:50:25.940237+0000","last_undegraded":"2023-10-17T01:50:25.940237+0000","last_fullsized":"2023-10-17T01:50:25.940237+0000","mapping_epoch":16,"log_start":"95'442417","ondisk_log_start":"95'442417","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":446,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 140s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":6841167872,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":451068,"num_read_kb":1774024,"num_write":450914,"num_write_kb":1803656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":3775,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:50:18.634677+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T01:50:18.634677+0000","last_peered":"2023-10-17T01:50:18.634677+0000","last_clean":"2023-10-17T01:50:18.634677+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:50:18.634677+0000","last_undegraded":"2023-10-17T01:50:18.634677+0000","last_fullsized":"2023-10-17T01:50:18.634677+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"97'441993","reported_seq":883379,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:50:26.389583+0000","last_change":"2023-10-17T01:50:24.948829+0000","last_active":"2023-10-17T01:50:26.389583+0000","last_peered":"2023-10-17T01:50:26.389583+0000","last_clean":"2023-10-17T01:50:26.389583+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:50:26.389583+0000","last_undegraded":"2023-10-17T01:50:26.389583+0000","last_fullsized":"2023-10-17T01:50:26.389583+0000","mapping_epoch":16,"log_start":"95'431916","ondisk_log_start":"95'431916","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6691393555,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":442474,"num_read_kb":1739664,"num_write":440720,"num_write_kb":1762874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"97'8690","reported_seq":12345,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:50:24.408931+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T01:50:24.408931+0000","last_peered":"2023-10-17T01:50:24.408931+0000","last_clean":"2023-10-17T01:50:24.408931+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:50:24.408931+0000","last_undegraded":"2023-10-17T01:50:24.408931+0000","last_fullsized":"2023-10-17T01:50:24.408931+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":8690,"log_dups_size":0,"ondisk_log_size":8690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":267,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-17T01:48:52.052685+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:48:52.052685+0000","last_peered":"2023-10-17T01:48:52.052685+0000","last_clean":"2023-10-17T01:48:52.052685+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:48:52.052685+0000","last_undegraded":"2023-10-17T01:48:52.052685+0000","last_fullsized":"2023-10-17T01:48:52.052685+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26915500051,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1776821,"num_read_kb":6987728,"num_write":1773517,"num_write_kb":7094062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14525001728,"data_stored":17925861414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40279,"ondisk_log_size":40279,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":42192,"ondisk_log_size":42192,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739191,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8478896,"kb_used_data":7091112,"kb_used_omap":562,"kb_used_meta":1387213,"kb_avail":85892944,"statfs":{"total":96636764160,"available":87954374656,"internally_reserved":0,"allocated":7261298688,"data_stored":8961155373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":575753,"internal_metadata":1420506871},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:50:04 2023","interfaces":[{"interface":"back","average":{"1min":0.29199999999999998,"5min":0.33300000000000002,"15min":0.36899999999999999},"min":{"1min":0.19,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.50900000000000001,"5min":2.2109999999999999,"15min":4.5030000000000001},"last":0.254},{"interface":"front","average":{"1min":0.311,"5min":0.44600000000000001,"15min":0.51900000000000002},"min":{"1min":0.20699999999999999,"5min":0.20300000000000001,"15min":0.19900000000000001},"max":{"1min":0.82199999999999995,"5min":4.2610000000000001,"15min":5.4240000000000004},"last":0.28899999999999998}]},{"osd":1,"last update":"Tue Oct 17 01:50:20 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.46800000000000003,"15min":0.48199999999999998},"min":{"1min":0.23100000000000001,"5min":0.19400000000000001,"15min":0.191},"max":{"1min":2.2000000000000002,"5min":4.3419999999999996,"15min":12.413},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.442,"5min":0.47299999999999998,"15min":0.48399999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":2.2309999999999999,"5min":4.1790000000000003,"15min":5.282},"last":0.30599999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739189,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5040336,"kb_used_data":3494024,"kb_used_omap":284,"kb_used_meta":1546019,"kb_avail":89331504,"statfs":{"total":96636764160,"available":91475460096,"internally_reserved":0,"allocated":3577880576,"data_stored":4415510860,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":291170,"internal_metadata":1583124126},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,5,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:49:49 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.64500000000000002,"15min":0.45400000000000001},"min":{"1min":0.17100000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":2.0089999999999999,"5min":23.177,"15min":23.177},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.69599999999999995,"15min":0.53400000000000003},"min":{"1min":0.23100000000000001,"5min":0.216,"15min":0.19700000000000001},"max":{"1min":2.0419999999999998,"5min":23.207999999999998,"15min":23.207999999999998},"last":0.32000000000000001}]},{"osd":2,"last update":"Tue Oct 17 01:49:49 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.41499999999999998,"15min":0.52400000000000002},"min":{"1min":0.21199999999999999,"5min":0.184,"15min":0.182},"max":{"1min":0.57999999999999996,"5min":5.1379999999999999,"15min":13.757},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.46100000000000002,"15min":0.60299999999999998},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.23100000000000001},"max":{"1min":0.61699999999999999,"5min":5.1799999999999997,"15min":14.195},"last":0.374}]}]},{"osd":0,"up_from":8,"seq":34359739189,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5355496,"kb_used_data":3601632,"kb_used_omap":289,"kb_used_meta":1753566,"kb_avail":89016344,"statfs":{"total":96636764160,"available":91152736256,"internally_reserved":0,"allocated":3688071168,"data_stored":4550546963,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":296403,"internal_metadata":1795652141},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,5,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:50:24 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.51300000000000001,"15min":0.60499999999999998},"min":{"1min":0.22900000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":1.5409999999999999,"5min":5.9829999999999997,"15min":15.317},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.56899999999999995,"15min":0.57699999999999996},"min":{"1min":0.20899999999999999,"5min":0.20100000000000001,"15min":0.184},"max":{"1min":4.9850000000000003,"5min":6.0750000000000002,"15min":7.5789999999999997},"last":0.51500000000000001}]},{"osd":2,"last update":"Tue Oct 17 01:50:24 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.32900000000000001,"15min":0.36499999999999999},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.18099999999999999},"max":{"1min":0.51700000000000002,"5min":2.2000000000000002,"15min":5.7709999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.53100000000000003,"15min":0.55900000000000005},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.19400000000000001},"max":{"1min":3.0630000000000002,"5min":6.1269999999999998,"15min":7.6230000000000002},"last":0.55400000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3687165952,"data_stored":4549763072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3577171968,"data_stored":4415004691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7260663808,"data_stored":8961093651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:50:28.776 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:50:28.776 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:50:28.777 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:50:28.777 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:50:28.778 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-17T01:50:29.424 INFO:teuthology.orchestra.run.smithi169.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-17T01:50:29.530 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-17T01:50:30.065 INFO:teuthology.orchestra.run.smithi169.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-17T01:50:30.094 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-17T01:50:30.634 INFO:teuthology.orchestra.run.smithi169.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-17T01:50:30.667 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-17T01:50:31.201 INFO:teuthology.orchestra.run.smithi169.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-17T01:50:31.241 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T01:50:47.615 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:50:47.601+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:50:47.623 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:50:47.617+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:50:47.623 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:50:47.617+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:50:47.630 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:50:47.625+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:50:47.630 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:50:47.625+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:50:47.630 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:50:47.625+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:50:51.242 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:50:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:50:51.795 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:50:51.847 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2157,"stamp":"2023-10-17T01:50:51.369518+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26948770648,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1808663,"num_read_kb":7039737,"num_write":1782192,"num_write_kb":7128419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82615,"ondisk_log_size":82615,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19168472,"kb_used_data":14254712,"kb_used_omap":1136,"kb_used_meta":4912591,"kb_avail":263947048,"statfs":{"total":289910292480,"available":270281777152,"internally_reserved":0,"allocated":14596825088,"data_stored":17996787852,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1163326,"internal_metadata":5030494146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,7,1,6,15,20,10,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18194432,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4370,"num_read_kb":17404,"num_write":4476,"num_write_kb":17904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002027"},"pg_stats":[{"pgid":"2.7","version":"97'11580","reported_seq":16550,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:50:44.757298+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T01:50:44.757298+0000","last_peered":"2023-10-17T01:50:44.757298+0000","last_clean":"2023-10-17T01:50:44.757298+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:50:44.757298+0000","last_undegraded":"2023-10-17T01:50:44.757298+0000","last_fullsized":"2023-10-17T01:50:44.757298+0000","mapping_epoch":13,"log_start":"52'1500","ondisk_log_start":"52'1500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"97'8723","reported_seq":13621,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:50:44.743260+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T01:50:44.743260+0000","last_peered":"2023-10-17T01:50:44.743260+0000","last_clean":"2023-10-17T01:50:44.743260+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:50:44.743260+0000","last_undegraded":"2023-10-17T01:50:44.743260+0000","last_fullsized":"2023-10-17T01:50:44.743260+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":8723,"log_dups_size":0,"ondisk_log_size":8723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"97'5849","reported_seq":8412,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:50:44.731810+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T01:50:44.731810+0000","last_peered":"2023-10-17T01:50:44.731810+0000","last_clean":"2023-10-17T01:50:44.731810+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:50:44.731810+0000","last_undegraded":"2023-10-17T01:50:44.731810+0000","last_fullsized":"2023-10-17T01:50:44.731810+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":5849,"log_dups_size":0,"ondisk_log_size":5849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"97'8754","reported_seq":13057,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:50:44.751365+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T01:50:44.751365+0000","last_peered":"2023-10-17T01:50:44.751365+0000","last_clean":"2023-10-17T01:50:44.751365+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:50:44.751365+0000","last_undegraded":"2023-10-17T01:50:44.751365+0000","last_fullsized":"2023-10-17T01:50:44.751365+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":8754,"log_dups_size":0,"ondisk_log_size":8754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"97'453114","reported_seq":903820,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:50:51.063690+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T01:50:51.063690+0000","last_peered":"2023-10-17T01:50:51.063690+0000","last_clean":"2023-10-17T01:50:51.063690+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:50:51.063690+0000","last_undegraded":"2023-10-17T01:50:51.063690+0000","last_fullsized":"2023-10-17T01:50:51.063690+0000","mapping_epoch":16,"log_start":"96'443108","ondisk_log_start":"96'443108","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":482,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 157s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":6830784512,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":451781,"num_read_kb":1776064,"num_write":451543,"num_write_kb":1806172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":247,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-17T01:49:15.415707+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:49:15.415707+0000","last_peered":"2023-10-17T01:49:15.415707+0000","last_clean":"2023-10-17T01:49:15.415707+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:49:15.415707+0000","last_undegraded":"2023-10-17T01:49:15.415707+0000","last_fullsized":"2023-10-17T01:49:15.415707+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"97'435931","reported_seq":870335,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:50:46.451217+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T01:50:46.451217+0000","last_peered":"2023-10-17T01:50:46.451217+0000","last_clean":"2023-10-17T01:50:46.451217+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:50:46.451217+0000","last_undegraded":"2023-10-17T01:50:46.451217+0000","last_fullsized":"2023-10-17T01:50:46.451217+0000","mapping_epoch":16,"log_start":"96'425917","ondisk_log_start":"96'425917","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":482,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 144s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6568697856,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":435427,"num_read_kb":1713616,"num_write":434411,"num_write_kb":1737644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2522,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:50:44.376921+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T01:50:44.376921+0000","last_peered":"2023-10-17T01:50:44.376921+0000","last_clean":"2023-10-17T01:50:44.376921+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:50:44.376921+0000","last_undegraded":"2023-10-17T01:50:44.376921+0000","last_fullsized":"2023-10-17T01:50:44.376921+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"97'454766","reported_seq":906961,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:50:51.062716+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T01:50:51.062716+0000","last_peered":"2023-10-17T01:50:51.062716+0000","last_clean":"2023-10-17T01:50:51.062716+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:50:51.062716+0000","last_undegraded":"2023-10-17T01:50:51.062716+0000","last_fullsized":"2023-10-17T01:50:51.062716+0000","mapping_epoch":16,"log_start":"96'444718","ondisk_log_start":"96'444718","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":532,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 165s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":6850326528,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":453268,"num_read_kb":1782768,"num_write":453174,"num_write_kb":1812696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":3791,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:50:44.268874+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T01:50:44.268874+0000","last_peered":"2023-10-17T01:50:44.268874+0000","last_clean":"2023-10-17T01:50:44.268874+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:50:44.268874+0000","last_undegraded":"2023-10-17T01:50:44.268874+0000","last_fullsized":"2023-10-17T01:50:44.268874+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"97'443707","reported_seq":886859,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:50:46.453450+0000","last_change":"2023-10-17T01:50:46.078564+0000","last_active":"2023-10-17T01:50:46.453450+0000","last_peered":"2023-10-17T01:50:46.453450+0000","last_clean":"2023-10-17T01:50:46.453450+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:50:46.453450+0000","last_undegraded":"2023-10-17T01:50:46.453450+0000","last_fullsized":"2023-10-17T01:50:46.453450+0000","mapping_epoch":16,"log_start":"96'433616","ondisk_log_start":"96'433616","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6698369043,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":444225,"num_read_kb":1746636,"num_write":442434,"num_write_kb":1769730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"97'8723","reported_seq":12390,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:50:44.784908+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T01:50:44.784908+0000","last_peered":"2023-10-17T01:50:44.784908+0000","last_clean":"2023-10-17T01:50:44.784908+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:50:44.784908+0000","last_undegraded":"2023-10-17T01:50:44.784908+0000","last_fullsized":"2023-10-17T01:50:44.784908+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":8723,"log_dups_size":0,"ondisk_log_size":8723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":268,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:49:57.117912+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:49:57.117912+0000","last_peered":"2023-10-17T01:49:57.117912+0000","last_clean":"2023-10-17T01:49:57.117912+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:49:57.117912+0000","last_undegraded":"2023-10-17T01:49:57.117912+0000","last_fullsized":"2023-10-17T01:49:57.117912+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26948177939,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1784701,"num_read_kb":7019084,"num_write":1781562,"num_write_kb":7126242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14594707456,"data_stored":17995567142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40159,"ondisk_log_size":40159,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":42357,"ondisk_log_size":42357,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739196,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8615420,"kb_used_data":7126708,"kb_used_omap":562,"kb_used_meta":1488141,"kb_avail":85756420,"statfs":{"total":96636764160,"available":87814574080,"internally_reserved":0,"allocated":7297748992,"data_stored":8997605677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":575753,"internal_metadata":1523857143},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,3,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:50:04 2023","interfaces":[{"interface":"back","average":{"1min":0.29199999999999998,"5min":0.33300000000000002,"15min":0.36899999999999999},"min":{"1min":0.19,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.50900000000000001,"5min":2.2109999999999999,"15min":4.5030000000000001},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.311,"5min":0.44600000000000001,"15min":0.51900000000000002},"min":{"1min":0.20699999999999999,"5min":0.20300000000000001,"15min":0.19900000000000001},"max":{"1min":0.82199999999999995,"5min":4.2610000000000001,"15min":5.4240000000000004},"last":0.28899999999999998}]},{"osd":1,"last update":"Tue Oct 17 01:50:20 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.46800000000000003,"15min":0.48199999999999998},"min":{"1min":0.23100000000000001,"5min":0.19400000000000001,"15min":0.191},"max":{"1min":2.2000000000000002,"5min":4.3419999999999996,"15min":12.413},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.442,"5min":0.47299999999999998,"15min":0.48399999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":2.2309999999999999,"5min":4.1790000000000003,"15min":5.282},"last":0.70499999999999996}]}]},{"osd":1,"up_from":8,"seq":34359739193,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5121056,"kb_used_data":3508180,"kb_used_omap":284,"kb_used_meta":1612579,"kb_avail":89250784,"statfs":{"total":96636764160,"available":91392802816,"internally_reserved":0,"allocated":3592376320,"data_stored":4430006604,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":291170,"internal_metadata":1651281566},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,3,7,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:49:49 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.64500000000000002,"15min":0.45400000000000001},"min":{"1min":0.17100000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":2.0089999999999999,"5min":23.177,"15min":23.177},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.69599999999999995,"15min":0.53400000000000003},"min":{"1min":0.23100000000000001,"5min":0.216,"15min":0.19700000000000001},"max":{"1min":2.0419999999999998,"5min":23.207999999999998,"15min":23.207999999999998},"last":0.316}]},{"osd":2,"last update":"Tue Oct 17 01:49:49 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.41499999999999998,"15min":0.52400000000000002},"min":{"1min":0.21199999999999999,"5min":0.184,"15min":0.182},"max":{"1min":0.57999999999999996,"5min":5.1379999999999999,"15min":13.757},"last":0.443},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.46100000000000002,"15min":0.60299999999999998},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.23100000000000001},"max":{"1min":0.61699999999999999,"5min":5.1799999999999997,"15min":14.195},"last":0.47099999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739194,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5431996,"kb_used_data":3619824,"kb_used_omap":289,"kb_used_meta":1811870,"kb_avail":88939844,"statfs":{"total":96636764160,"available":91074400256,"internally_reserved":0,"allocated":3706699776,"data_stored":4569175571,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":296403,"internal_metadata":1855355437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,1,0,6,11,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:50:24 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.51300000000000001,"15min":0.60499999999999998},"min":{"1min":0.22900000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":1.5409999999999999,"5min":5.9829999999999997,"15min":15.317},"last":0.17999999999999999},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.56899999999999995,"15min":0.57699999999999996},"min":{"1min":0.20899999999999999,"5min":0.20100000000000001,"15min":0.184},"max":{"1min":4.9850000000000003,"5min":6.0750000000000002,"15min":7.5789999999999997},"last":0.255}]},{"osd":2,"last update":"Tue Oct 17 01:50:24 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.32900000000000001,"15min":0.36499999999999999},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.18099999999999999},"max":{"1min":0.51700000000000002,"5min":2.2000000000000002,"15min":5.7709999999999999},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.53100000000000003,"15min":0.55900000000000005},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.19400000000000001},"max":{"1min":3.0630000000000002,"5min":6.1269999999999998,"15min":7.6230000000000002},"last":0.22500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3705737216,"data_stored":4568334336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3591557120,"data_stored":4429389843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7297413120,"data_stored":8997842963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:50:51.848 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:50:51.849 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:50:51.849 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:50:51.850 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:50:51.850 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T01:51:11.851 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:51:12.619 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:51:12.621 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:51:12.652 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2168,"stamp":"2023-10-17T01:51:11.372927+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26978011992,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1815965,"num_read_kb":7068877,"num_write":1789395,"num_write_kb":7157231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82740,"ondisk_log_size":82740,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19402312,"kb_used_data":14312240,"kb_used_omap":1136,"kb_used_meta":5088911,"kb_avail":263713208,"statfs":{"total":289910292480,"available":270042324992,"internally_reserved":0,"allocated":14655733760,"data_stored":18055688668,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1163326,"internal_metadata":5211045826},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,9,10,11,13,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14721024,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3682,"num_read_kb":14700,"num_write":3633,"num_write_kb":14532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001653"},"pg_stats":[{"pgid":"2.7","version":"97'11629","reported_seq":16615,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:51:04.703650+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T01:51:04.703650+0000","last_peered":"2023-10-17T01:51:04.703650+0000","last_clean":"2023-10-17T01:51:04.703650+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:51:04.703650+0000","last_undegraded":"2023-10-17T01:51:04.703650+0000","last_fullsized":"2023-10-17T01:51:04.703650+0000","mapping_epoch":13,"log_start":"54'1600","ondisk_log_start":"54'1600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"97'8762","reported_seq":13671,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:51:04.703488+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T01:51:04.703488+0000","last_peered":"2023-10-17T01:51:04.703488+0000","last_clean":"2023-10-17T01:51:04.703488+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:51:04.703488+0000","last_undegraded":"2023-10-17T01:51:04.703488+0000","last_fullsized":"2023-10-17T01:51:04.703488+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":8762,"log_dups_size":0,"ondisk_log_size":8762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"97'5875","reported_seq":8446,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:51:04.704035+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T01:51:04.704035+0000","last_peered":"2023-10-17T01:51:04.704035+0000","last_clean":"2023-10-17T01:51:04.704035+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:51:04.704035+0000","last_undegraded":"2023-10-17T01:51:04.704035+0000","last_fullsized":"2023-10-17T01:51:04.704035+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":5875,"log_dups_size":0,"ondisk_log_size":5875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"97'8793","reported_seq":13108,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:51:04.703344+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T01:51:04.703344+0000","last_peered":"2023-10-17T01:51:04.703344+0000","last_clean":"2023-10-17T01:51:04.703344+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:51:04.703344+0000","last_undegraded":"2023-10-17T01:51:04.703344+0000","last_fullsized":"2023-10-17T01:51:04.703344+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":8793,"log_dups_size":0,"ondisk_log_size":8793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"98'454982","reported_seq":907632,"reported_epoch":98,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:51:11.159722+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T01:51:11.159722+0000","last_peered":"2023-10-17T01:51:11.159722+0000","last_clean":"2023-10-17T01:51:11.159722+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:51:11.159722+0000","last_undegraded":"2023-10-17T01:51:11.159722+0000","last_fullsized":"2023-10-17T01:51:11.159722+0000","mapping_epoch":16,"log_start":"96'444908","ondisk_log_start":"96'444908","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":532,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 177s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":6838247424,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":453714,"num_read_kb":1783772,"num_write":453386,"num_write_kb":1813544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":248,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:49:57.095048+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:49:57.095048+0000","last_peered":"2023-10-17T01:49:57.095048+0000","last_clean":"2023-10-17T01:49:57.095048+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:49:57.095048+0000","last_undegraded":"2023-10-17T01:49:57.095048+0000","last_fullsized":"2023-10-17T01:49:57.095048+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"97'437751","reported_seq":873912,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:51:06.620384+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T01:51:06.620384+0000","last_peered":"2023-10-17T01:51:06.620384+0000","last_clean":"2023-10-17T01:51:06.620384+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:51:06.620384+0000","last_undegraded":"2023-10-17T01:51:06.620384+0000","last_fullsized":"2023-10-17T01:51:06.620384+0000","mapping_epoch":16,"log_start":"96'427717","ondisk_log_start":"96'427717","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":596,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 165s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6575812608,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":437180,"num_read_kb":1720620,"num_write":436162,"num_write_kb":1744648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2531,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:51:04.711437+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T01:51:04.711437+0000","last_peered":"2023-10-17T01:51:04.711437+0000","last_clean":"2023-10-17T01:51:04.711437+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:51:04.711437+0000","last_undegraded":"2023-10-17T01:51:04.711437+0000","last_fullsized":"2023-10-17T01:51:04.711437+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"98'456633","reported_seq":910697,"reported_epoch":98,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:51:11.172435+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T01:51:11.172435+0000","last_peered":"2023-10-17T01:51:11.172435+0000","last_clean":"2023-10-17T01:51:11.172435+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:51:11.172435+0000","last_undegraded":"2023-10-17T01:51:11.172435+0000","last_fullsized":"2023-10-17T01:51:11.172435+0000","mapping_epoch":16,"log_start":"96'446618","ondisk_log_start":"96'446618","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":594,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 186s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":6857777152,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":455127,"num_read_kb":1790192,"num_write":455005,"num_write_kb":1820020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":3804,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:51:04.711769+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T01:51:04.711769+0000","last_peered":"2023-10-17T01:51:04.711769+0000","last_clean":"2023-10-17T01:51:04.711769+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:51:04.711769+0000","last_undegraded":"2023-10-17T01:51:04.711769+0000","last_fullsized":"2023-10-17T01:51:04.711769+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"97'445485","reported_seq":890398,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:51:06.621811+0000","last_change":"2023-10-17T01:51:05.211144+0000","last_active":"2023-10-17T01:51:06.621811+0000","last_peered":"2023-10-17T01:51:06.621811+0000","last_clean":"2023-10-17T01:51:06.621811+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:51:06.621811+0000","last_undegraded":"2023-10-17T01:51:06.621811+0000","last_fullsized":"2023-10-17T01:51:06.621811+0000","mapping_epoch":16,"log_start":"96'435416","ondisk_log_start":"96'435416","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6705582099,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":445982,"num_read_kb":1753640,"num_write":444212,"num_write_kb":1776842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"97'8762","reported_seq":12440,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:51:04.703360+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T01:51:04.703360+0000","last_peered":"2023-10-17T01:51:04.703360+0000","last_clean":"2023-10-17T01:51:04.703360+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:51:04.703360+0000","last_undegraded":"2023-10-17T01:51:04.703360+0000","last_fullsized":"2023-10-17T01:51:04.703360+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":8762,"log_dups_size":0,"ondisk_log_size":8762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":268,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:49:57.117912+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:49:57.117912+0000","last_peered":"2023-10-17T01:49:57.117912+0000","last_clean":"2023-10-17T01:49:57.117912+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:49:57.117912+0000","last_undegraded":"2023-10-17T01:49:57.117912+0000","last_fullsized":"2023-10-17T01:49:57.117912+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26977419283,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1792003,"num_read_kb":7048224,"num_write":1788765,"num_write_kb":7155054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14653775872,"data_stored":18054635558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40192,"ondisk_log_size":40192,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":42449,"ondisk_log_size":42449,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739200,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8730236,"kb_used_data":7155444,"kb_used_omap":562,"kb_used_meta":1574221,"kb_avail":85641604,"statfs":{"total":96636764160,"available":87697002496,"internally_reserved":0,"allocated":7327174656,"data_stored":9027027413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":575753,"internal_metadata":1612003063},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,1,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:51:04 2023","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.33500000000000002,"15min":0.372},"min":{"1min":0.214,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.63300000000000001,"5min":2.2109999999999999,"15min":4.5030000000000001},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.47899999999999998,"15min":0.53100000000000003},"min":{"1min":0.221,"5min":0.20300000000000001,"15min":0.19900000000000001},"max":{"1min":2.6099999999999999,"5min":4.2610000000000001,"15min":5.4240000000000004},"last":0.27300000000000002}]},{"osd":1,"last update":"Tue Oct 17 01:50:20 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.46800000000000003,"15min":0.48199999999999998},"min":{"1min":0.23100000000000001,"5min":0.19400000000000001,"15min":0.191},"max":{"1min":2.2000000000000002,"5min":4.3419999999999996,"15min":12.413},"last":0.317},{"interface":"front","average":{"1min":0.442,"5min":0.47299999999999998,"15min":0.48399999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":2.2309999999999999,"5min":4.1790000000000003,"15min":5.282},"last":0.33800000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739197,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5189316,"kb_used_data":3522300,"kb_used_omap":284,"kb_used_meta":1666723,"kb_avail":89182524,"statfs":{"total":96636764160,"available":91322904576,"internally_reserved":0,"allocated":3606835200,"data_stored":4444465484,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":291170,"internal_metadata":1706725022},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,1,2,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:50:52 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.47299999999999998},"min":{"1min":0.183,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":5.3869999999999996,"5min":23.177,"15min":23.177},"last":0.251},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70799999999999996,"15min":0.55500000000000005},"min":{"1min":0.23999999999999999,"5min":0.216,"15min":0.19700000000000001},"max":{"1min":5.4379999999999997,"5min":23.207999999999998,"15min":23.207999999999998},"last":0.32800000000000001}]},{"osd":2,"last update":"Tue Oct 17 01:50:52 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.49099999999999999,"15min":0.54800000000000004},"min":{"1min":0.188,"5min":0.188,"15min":0.182},"max":{"1min":5.423,"5min":5.423,"15min":13.757},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.746,"5min":0.53500000000000003,"15min":0.628},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.23100000000000001},"max":{"1min":5.4569999999999999,"5min":5.4569999999999999,"15min":14.195},"last":0.36299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739198,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5482760,"kb_used_data":3634496,"kb_used_omap":289,"kb_used_meta":1847966,"kb_avail":88889080,"statfs":{"total":96636764160,"available":91022417920,"internally_reserved":0,"allocated":3721723904,"data_stored":4584195771,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":296403,"internal_metadata":1892317741},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,4,6,8,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:50:24 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.51300000000000001,"15min":0.60499999999999998},"min":{"1min":0.22900000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":1.5409999999999999,"5min":5.9829999999999997,"15min":15.317},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.56899999999999995,"15min":0.57699999999999996},"min":{"1min":0.20899999999999999,"5min":0.20100000000000001,"15min":0.184},"max":{"1min":4.9850000000000003,"5min":6.0750000000000002,"15min":7.5789999999999997},"last":0.22800000000000001}]},{"osd":2,"last update":"Tue Oct 17 01:50:24 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.32900000000000001,"15min":0.36499999999999999},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.18099999999999999},"max":{"1min":0.51700000000000002,"5min":2.2000000000000002,"15min":5.7709999999999999},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.53100000000000003,"15min":0.55900000000000005},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.19400000000000001},"max":{"1min":3.0630000000000002,"5min":6.1269999999999998,"15min":7.6230000000000002},"last":0.33300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3720777728,"data_stored":4583374848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3606024192,"data_stored":4443856915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7326973952,"data_stored":9027403795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:51:12.654 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:51:12.654 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:51:12.655 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:51:12.655 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:51:12.655 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T01:51:32.656 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:51:33.246 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:51:33.247 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:51:33.277 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2178,"stamp":"2023-10-17T01:51:31.376321+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27007228760,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1823232,"num_read_kb":7097841,"num_write":1796558,"num_write_kb":7185883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83076,"ondisk_log_size":83076,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19637032,"kb_used_data":14369296,"kb_used_omap":1136,"kb_used_meta":5266575,"kb_avail":263478488,"statfs":{"total":289910292480,"available":269801971712,"internally_reserved":0,"allocated":14714159104,"data_stored":18114110084,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1163326,"internal_metadata":5392973762},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,1,4,10,6,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18190336,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4551,"num_read_kb":18144,"num_write":4458,"num_write_kb":17832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002029"},"pg_stats":[{"pgid":"2.7","version":"98'11705","reported_seq":16708,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-17T01:51:25.665488+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T01:51:25.665488+0000","last_peered":"2023-10-17T01:51:25.665488+0000","last_clean":"2023-10-17T01:51:25.665488+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:51:25.665488+0000","last_undegraded":"2023-10-17T01:51:25.665488+0000","last_fullsized":"2023-10-17T01:51:25.665488+0000","mapping_epoch":13,"log_start":"54'1700","ondisk_log_start":"54'1700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"98'8819","reported_seq":13741,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-17T01:51:25.664055+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T01:51:25.664055+0000","last_peered":"2023-10-17T01:51:25.664055+0000","last_clean":"2023-10-17T01:51:25.664055+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:51:25.664055+0000","last_undegraded":"2023-10-17T01:51:25.664055+0000","last_fullsized":"2023-10-17T01:51:25.664055+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":8819,"log_dups_size":0,"ondisk_log_size":8819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"98'5913","reported_seq":8494,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-17T01:51:25.658143+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T01:51:25.658143+0000","last_peered":"2023-10-17T01:51:25.658143+0000","last_clean":"2023-10-17T01:51:25.658143+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:51:25.658143+0000","last_undegraded":"2023-10-17T01:51:25.658143+0000","last_fullsized":"2023-10-17T01:51:25.658143+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":5913,"log_dups_size":0,"ondisk_log_size":5913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"98'8850","reported_seq":13179,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-17T01:51:25.662265+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T01:51:25.662265+0000","last_peered":"2023-10-17T01:51:25.662265+0000","last_clean":"2023-10-17T01:51:25.662265+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:51:25.662265+0000","last_undegraded":"2023-10-17T01:51:25.662265+0000","last_fullsized":"2023-10-17T01:51:25.662265+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":8850,"log_dups_size":0,"ondisk_log_size":8850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"98'456791","reported_seq":911287,"reported_epoch":98,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:51:31.349381+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T01:51:31.349381+0000","last_peered":"2023-10-17T01:51:31.349381+0000","last_clean":"2023-10-17T01:51:31.349381+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:51:31.349381+0000","last_undegraded":"2023-10-17T01:51:31.349381+0000","last_fullsized":"2023-10-17T01:51:31.349381+0000","mapping_epoch":16,"log_start":"96'446709","ondisk_log_start":"96'446709","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":576,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 198s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":6845550592,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":455550,"num_read_kb":1791072,"num_write":455167,"num_write_kb":1820668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":249,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-17T01:51:07.104010+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:51:07.104010+0000","last_peered":"2023-10-17T01:51:07.104010+0000","last_clean":"2023-10-17T01:51:07.104010+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:51:07.104010+0000","last_undegraded":"2023-10-17T01:51:07.104010+0000","last_fullsized":"2023-10-17T01:51:07.104010+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"98'439507","reported_seq":877432,"reported_epoch":98,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:51:26.804016+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T01:51:26.804016+0000","last_peered":"2023-10-17T01:51:26.804016+0000","last_clean":"2023-10-17T01:51:26.804016+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:51:26.804016+0000","last_undegraded":"2023-10-17T01:51:26.804016+0000","last_fullsized":"2023-10-17T01:51:26.804016+0000","mapping_epoch":16,"log_start":"96'429417","ondisk_log_start":"96'429417","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":660,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 185s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6582804480,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":438924,"num_read_kb":1727576,"num_write":437884,"num_write_kb":1751536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2540,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-17T01:51:25.188626+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T01:51:25.188626+0000","last_peered":"2023-10-17T01:51:25.188626+0000","last_clean":"2023-10-17T01:51:25.188626+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:51:25.188626+0000","last_undegraded":"2023-10-17T01:51:25.188626+0000","last_fullsized":"2023-10-17T01:51:25.188626+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"98'458516","reported_seq":914445,"reported_epoch":98,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:51:31.342985+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T01:51:31.342985+0000","last_peered":"2023-10-17T01:51:31.342985+0000","last_clean":"2023-10-17T01:51:31.342985+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:51:31.342985+0000","last_undegraded":"2023-10-17T01:51:31.342985+0000","last_fullsized":"2023-10-17T01:51:31.342985+0000","mapping_epoch":16,"log_start":"96'448418","ondisk_log_start":"96'448418","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":644,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 206s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":6865321984,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":456980,"num_read_kb":1797592,"num_write":456861,"num_write_kb":1827444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":3816,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-17T01:51:25.218566+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T01:51:25.218566+0000","last_peered":"2023-10-17T01:51:25.218566+0000","last_clean":"2023-10-17T01:51:25.218566+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:51:25.218566+0000","last_undegraded":"2023-10-17T01:51:25.218566+0000","last_fullsized":"2023-10-17T01:51:25.218566+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"98'447289","reported_seq":894046,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-17T01:51:26.841510+0000","last_change":"2023-10-17T01:51:21.579013+0000","last_active":"2023-10-17T01:51:26.841510+0000","last_peered":"2023-10-17T01:51:26.841510+0000","last_clean":"2023-10-17T01:51:26.841510+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:51:26.841510+0000","last_undegraded":"2023-10-17T01:51:26.841510+0000","last_fullsized":"2023-10-17T01:51:26.841510+0000","mapping_epoch":16,"log_start":"96'437216","ondisk_log_start":"96'437216","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6712958995,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":447816,"num_read_kb":1760948,"num_write":446016,"num_write_kb":1784058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"98'8819","reported_seq":12511,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-17T01:51:25.673465+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T01:51:25.673465+0000","last_peered":"2023-10-17T01:51:25.673465+0000","last_clean":"2023-10-17T01:51:25.673465+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:51:25.673465+0000","last_undegraded":"2023-10-17T01:51:25.673465+0000","last_fullsized":"2023-10-17T01:51:25.673465+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":8819,"log_dups_size":0,"ondisk_log_size":8819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":269,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:50:36.455408+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:50:36.455408+0000","last_peered":"2023-10-17T01:50:36.455408+0000","last_clean":"2023-10-17T01:50:36.455408+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:50:36.455408+0000","last_undegraded":"2023-10-17T01:50:36.455408+0000","last_fullsized":"2023-10-17T01:50:36.455408+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27006636051,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1799270,"num_read_kb":7077188,"num_write":1795928,"num_write_kb":7183706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14712360960,"data_stored":18113220646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40343,"ondisk_log_size":40343,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":42634,"ondisk_log_size":42634,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739204,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8857112,"kb_used_data":7183760,"kb_used_omap":562,"kb_used_meta":1672781,"kb_avail":85514728,"statfs":{"total":96636764160,"available":87567081472,"internally_reserved":0,"allocated":7356170240,"data_stored":9056022997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":575753,"internal_metadata":1712928503},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,1,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:51:04 2023","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.33500000000000002,"15min":0.372},"min":{"1min":0.214,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.63300000000000001,"5min":2.2109999999999999,"15min":4.5030000000000001},"last":3.7250000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.47899999999999998,"15min":0.53100000000000003},"min":{"1min":0.221,"5min":0.20300000000000001,"15min":0.19900000000000001},"max":{"1min":2.6099999999999999,"5min":4.2610000000000001,"15min":5.4240000000000004},"last":3.153}]},{"osd":1,"last update":"Tue Oct 17 01:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.505,"15min":0.48899999999999999},"min":{"1min":0.21299999999999999,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":5.7270000000000003,"5min":5.7270000000000003,"15min":12.413},"last":0.221},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.56000000000000005,"15min":0.504},"min":{"1min":0.248,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":5.7599999999999998,"5min":5.7599999999999998,"15min":5.7599999999999998},"last":3.0070000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739201,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5239624,"kb_used_data":3536576,"kb_used_omap":284,"kb_used_meta":1702755,"kb_avail":89132216,"statfs":{"total":96636764160,"available":91271389184,"internally_reserved":0,"allocated":3621453824,"data_stored":4459080180,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":291170,"internal_metadata":1743621790},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,4,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:50:52 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.47299999999999998},"min":{"1min":0.183,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":5.3869999999999996,"5min":23.177,"15min":23.177},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70799999999999996,"15min":0.55500000000000005},"min":{"1min":0.23999999999999999,"5min":0.216,"15min":0.19700000000000001},"max":{"1min":5.4379999999999997,"5min":23.207999999999998,"15min":23.207999999999998},"last":0.32800000000000001}]},{"osd":2,"last update":"Tue Oct 17 01:50:52 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.49099999999999999,"15min":0.54800000000000004},"min":{"1min":0.188,"5min":0.188,"15min":0.182},"max":{"1min":5.423,"5min":5.423,"15min":13.757},"last":0.376},{"interface":"front","average":{"1min":0.746,"5min":0.53500000000000003,"15min":0.628},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.23100000000000001},"max":{"1min":5.4569999999999999,"5min":5.4569999999999999,"15min":14.195},"last":0.40600000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739202,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5540296,"kb_used_data":3648960,"kb_used_omap":289,"kb_used_meta":1891038,"kb_avail":88831544,"statfs":{"total":96636764160,"available":90963501056,"internally_reserved":0,"allocated":3736535040,"data_stored":4599006907,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":296403,"internal_metadata":1936423469},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.49399999999999999,"15min":0.60399999999999998},"min":{"1min":0.17999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.69999999999999996,"5min":5.9829999999999997,"15min":15.317},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.56599999999999995,"15min":0.57899999999999996},"min":{"1min":0.218,"5min":0.20100000000000001,"15min":0.184},"max":{"1min":0.77300000000000002,"5min":6.0750000000000002,"15min":7.5789999999999997},"last":0.39700000000000002}]},{"osd":2,"last update":"Tue Oct 17 01:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.32900000000000001,"15min":0.35799999999999998},"min":{"1min":0.22700000000000001,"5min":0.20100000000000001,"15min":0.18099999999999999},"max":{"1min":0.64200000000000002,"5min":2.2000000000000002,"15min":5.7709999999999999},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.51800000000000002,"15min":0.55900000000000005},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.83899999999999997,"5min":6.1269999999999998,"15min":7.6230000000000002},"last":0.35999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3735670784,"data_stored":4598267904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3620454400,"data_stored":4458287123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7356235776,"data_stored":9056665619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:51:33.279 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:51:33.280 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:51:33.280 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:51:33.280 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:51:33.281 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T01:51:53.283 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:51:53.833 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:51:53.834 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:51:53.872 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2189,"stamp":"2023-10-17T01:51:53.379979+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27040463704,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1831451,"num_read_kb":7130605,"num_write":1804706,"num_write_kb":7218475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83122,"ondisk_log_size":83122,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19894864,"kb_used_data":14430388,"kb_used_omap":1136,"kb_used_meta":5463311,"kb_avail":263220656,"statfs":{"total":289910292480,"available":269537951744,"internally_reserved":0,"allocated":14776717312,"data_stored":18176668292,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1163326,"internal_metadata":5594431426},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,8,3,16,15,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22196224,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5494,"num_read_kb":21904,"num_write":5457,"num_write_kb":21828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001979"},"pg_stats":[{"pgid":"2.7","version":"98'11777","reported_seq":16800,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-17T01:51:51.146468+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T01:51:51.146468+0000","last_peered":"2023-10-17T01:51:51.146468+0000","last_clean":"2023-10-17T01:51:51.146468+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:51:51.146468+0000","last_undegraded":"2023-10-17T01:51:51.146468+0000","last_fullsized":"2023-10-17T01:51:51.146468+0000","mapping_epoch":13,"log_start":"54'1700","ondisk_log_start":"54'1700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"98'8873","reported_seq":13811,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-17T01:51:51.139343+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T01:51:51.139343+0000","last_peered":"2023-10-17T01:51:51.139343+0000","last_clean":"2023-10-17T01:51:51.139343+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:51:51.139343+0000","last_undegraded":"2023-10-17T01:51:51.139343+0000","last_fullsized":"2023-10-17T01:51:51.139343+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":8873,"log_dups_size":0,"ondisk_log_size":8873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"98'5949","reported_seq":8540,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-17T01:51:51.123792+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T01:51:51.123792+0000","last_peered":"2023-10-17T01:51:51.123792+0000","last_clean":"2023-10-17T01:51:51.123792+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:51:51.123792+0000","last_undegraded":"2023-10-17T01:51:51.123792+0000","last_fullsized":"2023-10-17T01:51:51.123792+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":5949,"log_dups_size":0,"ondisk_log_size":5949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"98'8904","reported_seq":13248,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-17T01:51:51.140745+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T01:51:51.140745+0000","last_peered":"2023-10-17T01:51:51.140745+0000","last_clean":"2023-10-17T01:51:51.140745+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:51:51.140745+0000","last_undegraded":"2023-10-17T01:51:51.140745+0000","last_fullsized":"2023-10-17T01:51:51.140745+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":8904,"log_dups_size":0,"ondisk_log_size":8904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"98'458660","reported_seq":914991,"reported_epoch":98,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:51:51.482945+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T01:51:51.482945+0000","last_peered":"2023-10-17T01:51:51.482945+0000","last_clean":"2023-10-17T01:51:51.482945+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:51:51.482945+0000","last_undegraded":"2023-10-17T01:51:51.482945+0000","last_fullsized":"2023-10-17T01:51:51.482945+0000","mapping_epoch":16,"log_start":"97'448609","ondisk_log_start":"97'448609","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":644,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 218s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":6853062656,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":457376,"num_read_kb":1798348,"num_write":456998,"num_write_kb":1827992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":249,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-17T01:51:07.104010+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:51:07.104010+0000","last_peered":"2023-10-17T01:51:07.104010+0000","last_clean":"2023-10-17T01:51:07.104010+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:51:07.104010+0000","last_undegraded":"2023-10-17T01:51:07.104010+0000","last_fullsized":"2023-10-17T01:51:07.104010+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"98'441768","reported_seq":882016,"reported_epoch":98,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:51:52.000676+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T01:51:52.000676+0000","last_peered":"2023-10-17T01:51:52.000676+0000","last_clean":"2023-10-17T01:51:52.000676+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:51:52.000676+0000","last_undegraded":"2023-10-17T01:51:52.000676+0000","last_fullsized":"2023-10-17T01:51:52.000676+0000","mapping_epoch":16,"log_start":"97'431717","ondisk_log_start":"97'431717","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":746,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 210s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6591832064,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":441235,"num_read_kb":1736792,"num_write":440098,"num_write_kb":1760392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2548,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-17T01:51:45.664365+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T01:51:45.664365+0000","last_peered":"2023-10-17T01:51:45.664365+0000","last_clean":"2023-10-17T01:51:45.664365+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:51:45.664365+0000","last_undegraded":"2023-10-17T01:51:45.664365+0000","last_fullsized":"2023-10-17T01:51:45.664365+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"98'460333","reported_seq":918166,"reported_epoch":98,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:51:51.472586+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T01:51:51.472586+0000","last_peered":"2023-10-17T01:51:51.472586+0000","last_clean":"2023-10-17T01:51:51.472586+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:51:51.472586+0000","last_undegraded":"2023-10-17T01:51:51.472586+0000","last_fullsized":"2023-10-17T01:51:51.472586+0000","mapping_epoch":16,"log_start":"97'450318","ondisk_log_start":"97'450318","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":720,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 226s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":6872539136,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":458873,"num_read_kb":1805132,"num_write":458635,"num_write_kb":1834540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":3828,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-17T01:51:45.666564+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T01:51:45.666564+0000","last_peered":"2023-10-17T01:51:45.666564+0000","last_clean":"2023-10-17T01:51:45.666564+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:51:45.666564+0000","last_undegraded":"2023-10-17T01:51:45.666564+0000","last_fullsized":"2023-10-17T01:51:45.666564+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"98'449618","reported_seq":898574,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-17T01:51:51.984436+0000","last_change":"2023-10-17T01:51:45.671793+0000","last_active":"2023-10-17T01:51:51.984436+0000","last_peered":"2023-10-17T01:51:51.984436+0000","last_clean":"2023-10-17T01:51:51.984436+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:51:51.984436+0000","last_undegraded":"2023-10-17T01:51:51.984436+0000","last_fullsized":"2023-10-17T01:51:51.984436+0000","mapping_epoch":16,"log_start":"97'439616","ondisk_log_start":"97'439616","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6722437139,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":450005,"num_read_kb":1769680,"num_write":448345,"num_write_kb":1793374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"98'8873","reported_seq":12580,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-17T01:51:51.179212+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T01:51:51.179212+0000","last_peered":"2023-10-17T01:51:51.179212+0000","last_clean":"2023-10-17T01:51:51.179212+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:51:51.179212+0000","last_undegraded":"2023-10-17T01:51:51.179212+0000","last_fullsized":"2023-10-17T01:51:51.179212+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":8873,"log_dups_size":0,"ondisk_log_size":8873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":269,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:50:36.455408+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:50:36.455408+0000","last_peered":"2023-10-17T01:50:36.455408+0000","last_clean":"2023-10-17T01:50:36.455408+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:50:36.455408+0000","last_undegraded":"2023-10-17T01:50:36.455408+0000","last_fullsized":"2023-10-17T01:50:36.455408+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27039870995,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1807489,"num_read_kb":7109952,"num_write":1804076,"num_write_kb":7216298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14775631872,"data_stored":18176491558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40119,"ondisk_log_size":40119,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":42904,"ondisk_log_size":42904,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739208,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8954088,"kb_used_data":7212700,"kb_used_omap":562,"kb_used_meta":1740813,"kb_avail":85417752,"statfs":{"total":96636764160,"available":87467778048,"internally_reserved":0,"allocated":7385804800,"data_stored":9085657557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":575753,"internal_metadata":1782593271},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,2,5,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:51:04 2023","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.33500000000000002,"15min":0.372},"min":{"1min":0.214,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.63300000000000001,"5min":2.2109999999999999,"15min":4.5030000000000001},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.47899999999999998,"15min":0.53100000000000003},"min":{"1min":0.221,"5min":0.20300000000000001,"15min":0.19900000000000001},"max":{"1min":2.6099999999999999,"5min":4.2610000000000001,"15min":5.4240000000000004},"last":0.26500000000000001}]},{"osd":1,"last update":"Tue Oct 17 01:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.505,"15min":0.48899999999999999},"min":{"1min":0.21299999999999999,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":5.7270000000000003,"5min":5.7270000000000003,"15min":12.413},"last":0.311},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.56000000000000005,"15min":0.504},"min":{"1min":0.248,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":5.7599999999999998,"5min":5.7599999999999998,"15min":5.7599999999999998},"last":0.28199999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739206,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5336448,"kb_used_data":3554680,"kb_used_omap":284,"kb_used_meta":1781475,"kb_avail":89035392,"statfs":{"total":96636764160,"available":91172241408,"internally_reserved":0,"allocated":3639992320,"data_stored":4477618676,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":291170,"internal_metadata":1824231070},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,0,9,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:50:52 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.47299999999999998},"min":{"1min":0.183,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":5.3869999999999996,"5min":23.177,"15min":23.177},"last":0.18099999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70799999999999996,"15min":0.55500000000000005},"min":{"1min":0.23999999999999999,"5min":0.216,"15min":0.19700000000000001},"max":{"1min":5.4379999999999997,"5min":23.207999999999998,"15min":23.207999999999998},"last":0.255}]},{"osd":2,"last update":"Tue Oct 17 01:50:52 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.49099999999999999,"15min":0.54800000000000004},"min":{"1min":0.188,"5min":0.188,"15min":0.182},"max":{"1min":5.423,"5min":5.423,"15min":13.757},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.746,"5min":0.53500000000000003,"15min":0.628},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.23100000000000001},"max":{"1min":5.4569999999999999,"5min":5.4569999999999999,"15min":14.195},"last":0.29699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739206,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5604328,"kb_used_data":3663008,"kb_used_omap":289,"kb_used_meta":1941022,"kb_avail":88767512,"statfs":{"total":96636764160,"available":90897932288,"internally_reserved":0,"allocated":3750920192,"data_stored":4613392059,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":296403,"internal_metadata":1987607085},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,1,2,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.49399999999999999,"15min":0.60399999999999998},"min":{"1min":0.17999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.69999999999999996,"5min":5.9829999999999997,"15min":15.317},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.56599999999999995,"15min":0.57899999999999996},"min":{"1min":0.218,"5min":0.20100000000000001,"15min":0.184},"max":{"1min":0.77300000000000002,"5min":6.0750000000000002,"15min":7.5789999999999997},"last":0.59599999999999997}]},{"osd":2,"last update":"Tue Oct 17 01:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.32900000000000001,"15min":0.35799999999999998},"min":{"1min":0.22700000000000001,"5min":0.20100000000000001,"15min":0.18099999999999999},"max":{"1min":0.64200000000000002,"5min":2.2000000000000002,"15min":5.7709999999999999},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.51800000000000002,"15min":0.55900000000000005},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.83899999999999997,"5min":6.1269999999999998,"15min":7.6230000000000002},"last":0.55900000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3750449152,"data_stored":4613046272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3639070720,"data_stored":4476903443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7386112000,"data_stored":9086541843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:51:53.873 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:51:53.874 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:51:53.874 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:51:53.875 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:51:53.875 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T01:51:56.525 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:51:56.520+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:51:56.526 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:51:56.520+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:51:56.542 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:51:56.536+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:51:56.543 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:51:56.536+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:51:56.543 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:51:56.536+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:51:56.543 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:51:56.536+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:51:56.544 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:51:56.536+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:51:56.544 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:51:56.536+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:51:56.544 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:51:56.536+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:51:56.544 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:51:56.536+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:51:56.545 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:51:56.540+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:52:13.877 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:52:14.439 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:52:14.440 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:52:14.493 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2199,"stamp":"2023-10-17T01:52:13.383288+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27069864792,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1838755,"num_read_kb":7159673,"num_write":1811866,"num_write_kb":7247115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83234,"ondisk_log_size":83234,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19517104,"kb_used_data":14488216,"kb_used_omap":1136,"kb_used_meta":5027727,"kb_avail":263598416,"statfs":{"total":289910292480,"available":269924777984,"internally_reserved":0,"allocated":14835933184,"data_stored":18235884164,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1163326,"internal_metadata":5148393410},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,9,10,16,14,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22261760,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5557,"num_read_kb":22104,"num_write":5431,"num_write_kb":21724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002035"},"pg_stats":[{"pgid":"2.7","version":"98'11817","reported_seq":16857,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-17T01:52:11.267952+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T01:52:11.267952+0000","last_peered":"2023-10-17T01:52:11.267952+0000","last_clean":"2023-10-17T01:52:11.267952+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:52:11.267952+0000","last_undegraded":"2023-10-17T01:52:11.267952+0000","last_fullsized":"2023-10-17T01:52:11.267952+0000","mapping_epoch":13,"log_start":"54'1800","ondisk_log_start":"54'1800","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"98'8903","reported_seq":13853,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-17T01:52:11.268071+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T01:52:11.268071+0000","last_peered":"2023-10-17T01:52:11.268071+0000","last_clean":"2023-10-17T01:52:11.268071+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:52:11.268071+0000","last_undegraded":"2023-10-17T01:52:11.268071+0000","last_fullsized":"2023-10-17T01:52:11.268071+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":8903,"log_dups_size":0,"ondisk_log_size":8903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"98'5969","reported_seq":8568,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-17T01:52:11.267215+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T01:52:11.267215+0000","last_peered":"2023-10-17T01:52:11.267215+0000","last_clean":"2023-10-17T01:52:11.267215+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:52:11.267215+0000","last_undegraded":"2023-10-17T01:52:11.267215+0000","last_fullsized":"2023-10-17T01:52:11.267215+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":5969,"log_dups_size":0,"ondisk_log_size":5969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"98'8934","reported_seq":13290,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-17T01:52:11.269188+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T01:52:11.269188+0000","last_peered":"2023-10-17T01:52:11.269188+0000","last_clean":"2023-10-17T01:52:11.269188+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:52:11.269188+0000","last_undegraded":"2023-10-17T01:52:11.269188+0000","last_fullsized":"2023-10-17T01:52:11.269188+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":8934,"log_dups_size":0,"ondisk_log_size":8934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"98'460537","reported_seq":918735,"reported_epoch":98,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:52:11.639086+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T01:52:11.639086+0000","last_peered":"2023-10-17T01:52:11.639086+0000","last_clean":"2023-10-17T01:52:11.639086+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:52:11.639086+0000","last_undegraded":"2023-10-17T01:52:11.639086+0000","last_fullsized":"2023-10-17T01:52:11.639086+0000","mapping_epoch":16,"log_start":"97'450510","ondisk_log_start":"97'450510","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":694,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 238s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":6860611584,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":459237,"num_read_kb":1805764,"num_write":458845,"num_write_kb":1835380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":249,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-17T01:51:07.104010+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:51:07.104010+0000","last_peered":"2023-10-17T01:51:07.104010+0000","last_clean":"2023-10-17T01:51:07.104010+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:51:07.104010+0000","last_undegraded":"2023-10-17T01:51:07.104010+0000","last_fullsized":"2023-10-17T01:51:07.104010+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"98'443567","reported_seq":885649,"reported_epoch":98,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:52:12.129934+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T01:52:12.129934+0000","last_peered":"2023-10-17T01:52:12.129934+0000","last_clean":"2023-10-17T01:52:12.129934+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:52:12.129934+0000","last_undegraded":"2023-10-17T01:52:12.129934+0000","last_fullsized":"2023-10-17T01:52:12.129934+0000","mapping_epoch":16,"log_start":"97'433517","ondisk_log_start":"97'433517","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":812,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 230s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6599065600,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":443063,"num_read_kb":1744068,"num_write":441858,"num_write_kb":1767432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2556,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-17T01:52:06.252819+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T01:52:06.252819+0000","last_peered":"2023-10-17T01:52:06.252819+0000","last_clean":"2023-10-17T01:52:06.252819+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:52:06.252819+0000","last_undegraded":"2023-10-17T01:52:06.252819+0000","last_fullsized":"2023-10-17T01:52:06.252819+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"98'462193","reported_seq":921850,"reported_epoch":98,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:52:11.612510+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T01:52:11.612510+0000","last_peered":"2023-10-17T01:52:11.612510+0000","last_clean":"2023-10-17T01:52:11.612510+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:52:11.612510+0000","last_undegraded":"2023-10-17T01:52:11.612510+0000","last_fullsized":"2023-10-17T01:52:11.612510+0000","mapping_epoch":16,"log_start":"97'452118","ondisk_log_start":"97'452118","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":776,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 246s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":6880002048,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":460693,"num_read_kb":1812360,"num_write":460461,"num_write_kb":1841844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":3840,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-17T01:52:06.144700+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T01:52:06.144700+0000","last_peered":"2023-10-17T01:52:06.144700+0000","last_clean":"2023-10-17T01:52:06.144700+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:52:06.144700+0000","last_undegraded":"2023-10-17T01:52:06.144700+0000","last_fullsized":"2023-10-17T01:52:06.144700+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"98'451345","reported_seq":902098,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-17T01:52:12.132526+0000","last_change":"2023-10-17T01:52:05.808367+0000","last_active":"2023-10-17T01:52:12.132526+0000","last_peered":"2023-10-17T01:52:12.132526+0000","last_clean":"2023-10-17T01:52:12.132526+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:52:12.132526+0000","last_undegraded":"2023-10-17T01:52:12.132526+0000","last_fullsized":"2023-10-17T01:52:12.132526+0000","mapping_epoch":16,"log_start":"97'441316","ondisk_log_start":"97'441316","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6729592851,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":451800,"num_read_kb":1776828,"num_write":450072,"num_write_kb":1800282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"98'8903","reported_seq":12622,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-17T01:52:11.267295+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T01:52:11.267295+0000","last_peered":"2023-10-17T01:52:11.267295+0000","last_clean":"2023-10-17T01:52:11.267295+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:52:11.267295+0000","last_undegraded":"2023-10-17T01:52:11.267295+0000","last_fullsized":"2023-10-17T01:52:11.267295+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":8903,"log_dups_size":0,"ondisk_log_size":8903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":269,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-17T01:50:36.455408+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:50:36.455408+0000","last_peered":"2023-10-17T01:50:36.455408+0000","last_clean":"2023-10-17T01:50:36.455408+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:50:36.455408+0000","last_undegraded":"2023-10-17T01:50:36.455408+0000","last_fullsized":"2023-10-17T01:50:36.455408+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27069272083,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1814793,"num_read_kb":7139020,"num_write":1811236,"num_write_kb":7244938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14834098176,"data_stored":18234957862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40181,"ondisk_log_size":40181,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":42954,"ondisk_log_size":42954,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739212,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9063224,"kb_used_data":7241328,"kb_used_omap":562,"kb_used_meta":1821325,"kb_avail":85308616,"statfs":{"total":96636764160,"available":87356022784,"internally_reserved":0,"allocated":7415119872,"data_stored":9114972629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":575753,"internal_metadata":1865037559},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,8,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:52:05 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.42399999999999999,"15min":0.39000000000000001},"min":{"1min":0.19,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":4.2140000000000004,"5min":4.2140000000000004,"15min":4.5030000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.5,"5min":0.46000000000000002,"15min":0.51100000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":3.153,"5min":4.2610000000000001,"15min":5.4240000000000004},"last":0.79100000000000004}]},{"osd":1,"last update":"Tue Oct 17 01:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.505,"15min":0.48899999999999999},"min":{"1min":0.21299999999999999,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":5.7270000000000003,"5min":5.7270000000000003,"15min":12.413},"last":0.20699999999999999},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.56000000000000005,"15min":0.504},"min":{"1min":0.248,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":5.7599999999999998,"5min":5.7599999999999998,"15min":5.7599999999999998},"last":0.24399999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739210,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5429512,"kb_used_data":3568640,"kb_used_omap":284,"kb_used_meta":1860579,"kb_avail":88942328,"statfs":{"total":96636764160,"available":91076943872,"internally_reserved":0,"allocated":3654287360,"data_stored":4491913716,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":291170,"internal_metadata":1905233566},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,7,5,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:51:53 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.72399999999999998,"15min":0.498},"min":{"1min":0.18099999999999999,"5min":0.17100000000000001,"15min":0.16200000000000001},"max":{"1min":4.9109999999999996,"5min":23.177,"15min":23.177},"last":0.312},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.78000000000000003,"15min":0.57899999999999996},"min":{"1min":0.255,"5min":0.23100000000000001,"15min":0.19700000000000001},"max":{"1min":4.9500000000000002,"5min":23.207999999999998,"15min":23.207999999999998},"last":0.34699999999999998}]},{"osd":2,"last update":"Tue Oct 17 01:51:53 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.498,"15min":0.56399999999999995},"min":{"1min":0.19700000000000001,"5min":0.188,"15min":0.184},"max":{"1min":4.1050000000000004,"5min":5.423,"15min":13.757},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.54200000000000004,"15min":0.64300000000000002},"min":{"1min":0.254,"5min":0.24199999999999999,"15min":0.24099999999999999},"max":{"1min":4.1589999999999998,"5min":5.4569999999999999,"15min":14.195},"last":0.29299999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739210,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5024368,"kb_used_data":3678248,"kb_used_omap":289,"kb_used_meta":1345822,"kb_avail":89347472,"statfs":{"total":96636764160,"available":91491811328,"internally_reserved":0,"allocated":3766525952,"data_stored":4628997819,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":296403,"internal_metadata":1378122285},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,0,3,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.49399999999999999,"15min":0.60399999999999998},"min":{"1min":0.17999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.69999999999999996,"5min":5.9829999999999997,"15min":15.317},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.56599999999999995,"15min":0.57899999999999996},"min":{"1min":0.218,"5min":0.20100000000000001,"15min":0.184},"max":{"1min":0.77300000000000002,"5min":6.0750000000000002,"15min":7.5789999999999997},"last":0.317}]},{"osd":2,"last update":"Tue Oct 17 01:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.32900000000000001,"15min":0.35799999999999998},"min":{"1min":0.22700000000000001,"5min":0.20100000000000001,"15min":0.18099999999999999},"max":{"1min":0.64200000000000002,"5min":2.2000000000000002,"15min":5.7709999999999999},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.51800000000000002,"15min":0.55900000000000005},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.83899999999999997,"5min":6.1269999999999998,"15min":7.6230000000000002},"last":0.60699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3765485568,"data_stored":4628082688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3653341184,"data_stored":4491173907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7415271424,"data_stored":9115701267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:52:14.494 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:52:14.495 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:52:14.495 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:52:14.496 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:52:14.496 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T01:52:27.241 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:52:27.235+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:52:27.242 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:52:27.235+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:52:27.243 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:52:27.235+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbda750 2023-10-17T01:52:27.243 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:52:27.235+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:52:27.244 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:52:27.235+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:52:27.244 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:52:27.235+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:52:27.245 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:52:27.235+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:52:27.245 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:52:27.235+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:52:27.246 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:52:27.235+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:52:27.247 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:52:27.235+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:52:27.247 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:52:27.235+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:52:27.248 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:52:27.235+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:52:27.248 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:52:27.235+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:52:27.249 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:52:27.235+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:52:27.250 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:52:27.235+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:52:34.497 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:52:35.132 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:52:35.132 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:52:35.178 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2210,"stamp":"2023-10-17T01:52:33.386715+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27099532120,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1845949,"num_read_kb":7188297,"num_write":1819037,"num_write_kb":7275799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83486,"ondisk_log_size":83486,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19773232,"kb_used_data":14546648,"kb_used_omap":1136,"kb_used_meta":5225423,"kb_avail":263342288,"statfs":{"total":289910292480,"available":269662502912,"internally_reserved":0,"allocated":14895767552,"data_stored":18295706619,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1163326,"internal_metadata":5350834114},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,5,4,7,12,14,20,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14827520,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3538,"num_read_kb":14056,"num_write":3599,"num_write_kb":14396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001683"},"pg_stats":[{"pgid":"2.7","version":"99'11901","reported_seq":16958,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-17T01:52:31.743717+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T01:52:31.743717+0000","last_peered":"2023-10-17T01:52:31.743717+0000","last_clean":"2023-10-17T01:52:31.743717+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:52:31.743717+0000","last_undegraded":"2023-10-17T01:52:31.743717+0000","last_fullsized":"2023-10-17T01:52:31.743717+0000","mapping_epoch":13,"log_start":"56'1900","ondisk_log_start":"56'1900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"99'8966","reported_seq":13929,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-17T01:52:31.743655+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T01:52:31.743655+0000","last_peered":"2023-10-17T01:52:31.743655+0000","last_clean":"2023-10-17T01:52:31.743655+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:52:31.743655+0000","last_undegraded":"2023-10-17T01:52:31.743655+0000","last_fullsized":"2023-10-17T01:52:31.743655+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":8966,"log_dups_size":0,"ondisk_log_size":8966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"99'6011","reported_seq":8619,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-17T01:52:31.743132+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T01:52:31.743132+0000","last_peered":"2023-10-17T01:52:31.743132+0000","last_clean":"2023-10-17T01:52:31.743132+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:52:31.743132+0000","last_undegraded":"2023-10-17T01:52:31.743132+0000","last_fullsized":"2023-10-17T01:52:31.743132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6011,"log_dups_size":0,"ondisk_log_size":6011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"99'8997","reported_seq":13366,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-17T01:52:31.743280+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T01:52:31.743280+0000","last_peered":"2023-10-17T01:52:31.743280+0000","last_clean":"2023-10-17T01:52:31.743280+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:52:31.743280+0000","last_undegraded":"2023-10-17T01:52:31.743280+0000","last_fullsized":"2023-10-17T01:52:31.743280+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":8997,"log_dups_size":0,"ondisk_log_size":8997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"99'462376","reported_seq":922381,"reported_epoch":99,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:52:31.772963+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T01:52:31.772963+0000","last_peered":"2023-10-17T01:52:31.772963+0000","last_clean":"2023-10-17T01:52:31.772963+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:52:31.772963+0000","last_undegraded":"2023-10-17T01:52:31.772963+0000","last_fullsized":"2023-10-17T01:52:31.772963+0000","mapping_epoch":16,"log_start":"97'452310","ondisk_log_start":"97'452310","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":720,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 258s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":6868189184,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":461036,"num_read_kb":1812916,"num_write":460671,"num_write_kb":1842684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":250,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-17T01:51:21.294145+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:51:21.294145+0000","last_peered":"2023-10-17T01:51:21.294145+0000","last_clean":"2023-10-17T01:51:21.294145+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:51:21.294145+0000","last_undegraded":"2023-10-17T01:51:21.294145+0000","last_fullsized":"2023-10-17T01:51:21.294145+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"99'445368","reported_seq":889199,"reported_epoch":99,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:52:32.266929+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T01:52:32.266929+0000","last_peered":"2023-10-17T01:52:32.266929+0000","last_clean":"2023-10-17T01:52:32.266929+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:52:32.266929+0000","last_undegraded":"2023-10-17T01:52:32.266929+0000","last_fullsized":"2023-10-17T01:52:32.266929+0000","mapping_epoch":16,"log_start":"97'435317","ondisk_log_start":"97'435317","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":870,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 250s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6606319616,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":444798,"num_read_kb":1750984,"num_write":443627,"num_write_kb":1774508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2565,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-17T01:52:26.630665+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T01:52:26.630665+0000","last_peered":"2023-10-17T01:52:26.630665+0000","last_clean":"2023-10-17T01:52:26.630665+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:52:26.630665+0000","last_undegraded":"2023-10-17T01:52:26.630665+0000","last_fullsized":"2023-10-17T01:52:26.630665+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"99'463974","reported_seq":925462,"reported_epoch":99,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:52:31.734562+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T01:52:31.734562+0000","last_peered":"2023-10-17T01:52:31.734562+0000","last_clean":"2023-10-17T01:52:31.734562+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:52:31.734562+0000","last_undegraded":"2023-10-17T01:52:31.734562+0000","last_fullsized":"2023-10-17T01:52:31.734562+0000","mapping_epoch":16,"log_start":"97'453918","ondisk_log_start":"97'453918","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":812,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 266s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":6887251968,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":462513,"num_read_kb":1819608,"num_write":462221,"num_write_kb":1848884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":3853,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-17T01:52:26.628971+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T01:52:26.628971+0000","last_peered":"2023-10-17T01:52:26.628971+0000","last_clean":"2023-10-17T01:52:26.628971+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:52:26.628971+0000","last_undegraded":"2023-10-17T01:52:26.628971+0000","last_fullsized":"2023-10-17T01:52:26.628971+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"99'453161","reported_seq":905763,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-17T01:52:32.264726+0000","last_change":"2023-10-17T01:52:30.202472+0000","last_active":"2023-10-17T01:52:32.264726+0000","last_peered":"2023-10-17T01:52:32.264726+0000","last_clean":"2023-10-17T01:52:32.264726+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:52:32.264726+0000","last_undegraded":"2023-10-17T01:52:32.264726+0000","last_fullsized":"2023-10-17T01:52:32.264726+0000","mapping_epoch":16,"log_start":"97'443116","ondisk_log_start":"97'443116","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6737178643,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":453640,"num_read_kb":1784136,"num_write":451888,"num_write_kb":1807546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"99'8966","reported_seq":12698,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-17T01:52:31.743292+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T01:52:31.743292+0000","last_peered":"2023-10-17T01:52:31.743292+0000","last_clean":"2023-10-17T01:52:31.743292+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:52:31.743292+0000","last_undegraded":"2023-10-17T01:52:31.743292+0000","last_fullsized":"2023-10-17T01:52:31.743292+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":8966,"log_dups_size":0,"ondisk_log_size":8966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":270,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-17T01:51:07.111969+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:51:07.111969+0000","last_peered":"2023-10-17T01:51:07.111969+0000","last_clean":"2023-10-17T01:51:07.111969+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:51:07.111969+0000","last_undegraded":"2023-10-17T01:51:07.111969+0000","last_fullsized":"2023-10-17T01:51:07.111969+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27098939411,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1821987,"num_read_kb":7167644,"num_write":1818407,"num_write_kb":7273622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14892838912,"data_stored":18293698598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40218,"ondisk_log_size":40218,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":43169,"ondisk_log_size":43169,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739216,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9218472,"kb_used_data":7271072,"kb_used_omap":562,"kb_used_meta":1946829,"kb_avail":85153368,"statfs":{"total":96636764160,"available":87197048832,"internally_reserved":0,"allocated":7445577728,"data_stored":9145426514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":575753,"internal_metadata":1993553655},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,4,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:52:05 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.42399999999999999,"15min":0.39000000000000001},"min":{"1min":0.19,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":4.2140000000000004,"5min":4.2140000000000004,"15min":4.5030000000000001},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.5,"5min":0.46000000000000002,"15min":0.51100000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":3.153,"5min":4.2610000000000001,"15min":5.4240000000000004},"last":0.26200000000000001}]},{"osd":1,"last update":"Tue Oct 17 01:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.505,"15min":0.48899999999999999},"min":{"1min":0.21299999999999999,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":5.7270000000000003,"5min":5.7270000000000003,"15min":12.413},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.56000000000000005,"15min":0.504},"min":{"1min":0.248,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":5.7599999999999998,"5min":5.7599999999999998,"15min":5.7599999999999998},"last":0.32300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739214,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5461948,"kb_used_data":3583028,"kb_used_omap":284,"kb_used_meta":1878627,"kb_avail":88909892,"statfs":{"total":96636764160,"available":91043729408,"internally_reserved":0,"allocated":3669020672,"data_stored":4506643057,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":291170,"internal_metadata":1923714718},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,2,1,4,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:51:53 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.72399999999999998,"15min":0.498},"min":{"1min":0.18099999999999999,"5min":0.17100000000000001,"15min":0.16200000000000001},"max":{"1min":4.9109999999999996,"5min":23.177,"15min":23.177},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.78000000000000003,"15min":0.57899999999999996},"min":{"1min":0.255,"5min":0.23100000000000001,"15min":0.19700000000000001},"max":{"1min":4.9500000000000002,"5min":23.207999999999998,"15min":23.207999999999998},"last":0.307}]},{"osd":2,"last update":"Tue Oct 17 01:51:53 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.498,"15min":0.56399999999999995},"min":{"1min":0.19700000000000001,"5min":0.188,"15min":0.184},"max":{"1min":4.1050000000000004,"5min":5.423,"15min":13.757},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.54200000000000004,"15min":0.64300000000000002},"min":{"1min":0.254,"5min":0.24199999999999999,"15min":0.24099999999999999},"max":{"1min":4.1589999999999998,"5min":5.4569999999999999,"15min":14.195},"last":0.35899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739214,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5092812,"kb_used_data":3692548,"kb_used_omap":289,"kb_used_meta":1399966,"kb_avail":89279028,"statfs":{"total":96636764160,"available":91421724672,"internally_reserved":0,"allocated":3781169152,"data_stored":4643637048,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":296403,"internal_metadata":1433565741},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,5,7,16,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:52:31 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.496,"15min":0.53900000000000003},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.69899999999999995,"5min":5.9829999999999997,"15min":15.317},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.626,"15min":0.57499999999999996},"min":{"1min":0.23000000000000001,"5min":0.20100000000000001,"15min":0.184},"max":{"1min":3.734,"5min":6.0750000000000002,"15min":7.5789999999999997},"last":0.60999999999999999}]},{"osd":2,"last update":"Tue Oct 17 01:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.32900000000000001,"15min":0.35799999999999998},"min":{"1min":0.22700000000000001,"5min":0.20100000000000001,"15min":0.18099999999999999},"max":{"1min":0.64200000000000002,"5min":2.2000000000000002,"15min":5.7709999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.51800000000000002,"15min":0.55900000000000005},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.83899999999999997,"5min":6.1269999999999998,"15min":7.6230000000000002},"last":2.1819999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3780173824,"data_stored":4642770944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3668025344,"data_stored":4505858067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7444639744,"data_stored":9145069587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:52:35.180 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:52:35.180 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:52:35.180 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:52:35.181 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:52:35.181 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-17T01:52:35.878 INFO:teuthology.orchestra.run.smithi169.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-17T01:52:35.918 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-17T01:52:36.542 INFO:teuthology.orchestra.run.smithi169.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-17T01:52:36.573 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-17T01:52:37.124 INFO:teuthology.orchestra.run.smithi169.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-17T01:52:37.158 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-17T01:52:37.774 INFO:teuthology.orchestra.run.smithi169.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-17T01:52:37.815 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T01:52:57.816 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:52:58.382 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:52:58.383 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:52:58.430 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2222,"stamp":"2023-10-17T01:52:57.391081+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27132312408,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1853915,"num_read_kb":7220021,"num_write":1827018,"num_write_kb":7307723,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83782,"ondisk_log_size":83782,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20089364,"kb_used_data":14612084,"kb_used_omap":1136,"kb_used_meta":5476111,"kb_avail":263026156,"statfs":{"total":289910292480,"available":269338783744,"internally_reserved":0,"allocated":14962774016,"data_stored":18362713083,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1163326,"internal_metadata":5607538626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,5,9,12,13,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18071552,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4414,"num_read_kb":17608,"num_write":4399,"num_write_kb":17596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001943"},"pg_stats":[{"pgid":"2.7","version":"99'11957","reported_seq":17030,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-17T01:52:52.224242+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T01:52:52.224242+0000","last_peered":"2023-10-17T01:52:52.224242+0000","last_clean":"2023-10-17T01:52:52.224242+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:52:52.224242+0000","last_undegraded":"2023-10-17T01:52:52.224242+0000","last_fullsized":"2023-10-17T01:52:52.224242+0000","mapping_epoch":13,"log_start":"56'1900","ondisk_log_start":"56'1900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"99'9008","reported_seq":13983,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-17T01:52:52.223904+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T01:52:52.223904+0000","last_peered":"2023-10-17T01:52:52.223904+0000","last_clean":"2023-10-17T01:52:52.223904+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:52:52.223904+0000","last_undegraded":"2023-10-17T01:52:52.223904+0000","last_fullsized":"2023-10-17T01:52:52.223904+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":9008,"log_dups_size":0,"ondisk_log_size":9008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"99'6039","reported_seq":8655,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-17T01:52:52.223155+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T01:52:52.223155+0000","last_peered":"2023-10-17T01:52:52.223155+0000","last_clean":"2023-10-17T01:52:52.223155+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:52:52.223155+0000","last_undegraded":"2023-10-17T01:52:52.223155+0000","last_fullsized":"2023-10-17T01:52:52.223155+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6039,"log_dups_size":0,"ondisk_log_size":6039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"99'9039","reported_seq":13420,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-17T01:52:52.223619+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T01:52:52.223619+0000","last_peered":"2023-10-17T01:52:52.223619+0000","last_clean":"2023-10-17T01:52:52.223619+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:52:52.223619+0000","last_undegraded":"2023-10-17T01:52:52.223619+0000","last_fullsized":"2023-10-17T01:52:52.223619+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":9039,"log_dups_size":0,"ondisk_log_size":9039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"99'464641","reported_seq":926900,"reported_epoch":99,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:52:56.974892+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T01:52:56.974892+0000","last_peered":"2023-10-17T01:52:56.974892+0000","last_clean":"2023-10-17T01:52:56.974892+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:52:56.974892+0000","last_undegraded":"2023-10-17T01:52:56.974892+0000","last_fullsized":"2023-10-17T01:52:56.974892+0000","mapping_epoch":16,"log_start":"98'454610","ondisk_log_start":"98'454610","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":766,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 283s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":6877478912,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":463274,"num_read_kb":1821808,"num_write":462909,"num_write_kb":1851636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":250,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-17T01:51:21.294145+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:51:21.294145+0000","last_peered":"2023-10-17T01:51:21.294145+0000","last_clean":"2023-10-17T01:51:21.294145+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:51:21.294145+0000","last_undegraded":"2023-10-17T01:51:21.294145+0000","last_fullsized":"2023-10-17T01:51:21.294145+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"99'447112","reported_seq":892687,"reported_epoch":99,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:52:52.372367+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T01:52:52.372367+0000","last_peered":"2023-10-17T01:52:52.372367+0000","last_clean":"2023-10-17T01:52:52.372367+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:52:52.372367+0000","last_undegraded":"2023-10-17T01:52:52.372367+0000","last_fullsized":"2023-10-17T01:52:52.372367+0000","mapping_epoch":16,"log_start":"97'437017","ondisk_log_start":"97'437017","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":932,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 270s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6613270528,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":446530,"num_read_kb":1757904,"num_write":445337,"num_write_kb":1781348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2575,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-17T01:52:52.224377+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T01:52:52.224377+0000","last_peered":"2023-10-17T01:52:52.224377+0000","last_clean":"2023-10-17T01:52:52.224377+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:52:52.224377+0000","last_undegraded":"2023-10-17T01:52:52.224377+0000","last_fullsized":"2023-10-17T01:52:52.224377+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"99'466196","reported_seq":929914,"reported_epoch":99,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:52:56.973689+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T01:52:56.973689+0000","last_peered":"2023-10-17T01:52:56.973689+0000","last_clean":"2023-10-17T01:52:56.973689+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:52:56.973689+0000","last_undegraded":"2023-10-17T01:52:56.973689+0000","last_fullsized":"2023-10-17T01:52:56.973689+0000","mapping_epoch":16,"log_start":"97'456118","ondisk_log_start":"97'456118","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":888,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 291s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":6896160768,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":464724,"num_read_kb":1828412,"num_write":464399,"num_write_kb":1857596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":3869,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-17T01:52:52.223872+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T01:52:52.223872+0000","last_peered":"2023-10-17T01:52:52.223872+0000","last_clean":"2023-10-17T01:52:52.223872+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:52:52.223872+0000","last_undegraded":"2023-10-17T01:52:52.223872+0000","last_fullsized":"2023-10-17T01:52:52.223872+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"99'455016","reported_seq":909413,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-17T01:52:52.388724+0000","last_change":"2023-10-17T01:52:51.968949+0000","last_active":"2023-10-17T01:52:52.388724+0000","last_peered":"2023-10-17T01:52:52.388724+0000","last_clean":"2023-10-17T01:52:52.388724+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:52:52.388724+0000","last_undegraded":"2023-10-17T01:52:52.388724+0000","last_fullsized":"2023-10-17T01:52:52.388724+0000","mapping_epoch":16,"log_start":"97'444916","ondisk_log_start":"97'444916","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6744809491,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":455425,"num_read_kb":1791244,"num_write":453743,"num_write_kb":1814966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"99'9008","reported_seq":12752,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-17T01:52:52.223237+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T01:52:52.223237+0000","last_peered":"2023-10-17T01:52:52.223237+0000","last_clean":"2023-10-17T01:52:52.223237+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:52:52.223237+0000","last_undegraded":"2023-10-17T01:52:52.223237+0000","last_fullsized":"2023-10-17T01:52:52.223237+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":9008,"log_dups_size":0,"ondisk_log_size":9008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":271,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-17T01:52:27.120223+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:52:27.120223+0000","last_peered":"2023-10-17T01:52:27.120223+0000","last_clean":"2023-10-17T01:52:27.120223+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:52:27.120223+0000","last_undegraded":"2023-10-17T01:52:27.120223+0000","last_fullsized":"2023-10-17T01:52:27.120223+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27131719699,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1829953,"num_read_kb":7199368,"num_write":1826388,"num_write_kb":7305546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14961725440,"data_stored":18362585126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40304,"ondisk_log_size":40304,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":43379,"ondisk_log_size":43379,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739221,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9374424,"kb_used_data":7305292,"kb_used_omap":562,"kb_used_meta":2068557,"kb_avail":84997416,"statfs":{"total":96636764160,"available":87037353984,"internally_reserved":0,"allocated":7480619008,"data_stored":9180467794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":575753,"internal_metadata":2118203127},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:52:05 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.42399999999999999,"15min":0.39000000000000001},"min":{"1min":0.19,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":4.2140000000000004,"5min":4.2140000000000004,"15min":4.5030000000000001},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.46000000000000002,"15min":0.51100000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":3.153,"5min":4.2610000000000001,"15min":5.4240000000000004},"last":3.5870000000000002}]},{"osd":1,"last update":"Tue Oct 17 01:52:30 2023","interfaces":[{"interface":"back","average":{"1min":0.28799999999999998,"5min":0.46800000000000003,"15min":0.48799999999999999},"min":{"1min":0.20699999999999999,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":0.37,"5min":5.7270000000000003,"15min":12.413},"last":3.4129999999999998},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.55600000000000005,"15min":0.51400000000000001},"min":{"1min":0.24399999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":3.0070000000000001,"5min":5.7599999999999998,"15min":5.7599999999999998},"last":3.4449999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739218,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5544052,"kb_used_data":3597160,"kb_used_omap":284,"kb_used_meta":1946595,"kb_avail":88827788,"statfs":{"total":96636764160,"available":90959654912,"internally_reserved":0,"allocated":3683491840,"data_stored":4521114225,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":291170,"internal_metadata":1993313950},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,4,6,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:51:53 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.72399999999999998,"15min":0.498},"min":{"1min":0.18099999999999999,"5min":0.17100000000000001,"15min":0.16200000000000001},"max":{"1min":4.9109999999999996,"5min":23.177,"15min":23.177},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.78000000000000003,"15min":0.57899999999999996},"min":{"1min":0.255,"5min":0.23100000000000001,"15min":0.19700000000000001},"max":{"1min":4.9500000000000002,"5min":23.207999999999998,"15min":23.207999999999998},"last":0.44900000000000001}]},{"osd":2,"last update":"Tue Oct 17 01:51:53 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.498,"15min":0.56399999999999995},"min":{"1min":0.19700000000000001,"5min":0.188,"15min":0.184},"max":{"1min":4.1050000000000004,"5min":5.423,"15min":13.757},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.54200000000000004,"15min":0.64300000000000002},"min":{"1min":0.254,"5min":0.24199999999999999,"15min":0.24099999999999999},"max":{"1min":4.1589999999999998,"5min":5.4569999999999999,"15min":14.195},"last":0.39300000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739219,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5170888,"kb_used_data":3709632,"kb_used_omap":289,"kb_used_meta":1460958,"kb_avail":89200952,"statfs":{"total":96636764160,"available":91341774848,"internally_reserved":0,"allocated":3798663168,"data_stored":4661131064,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":296403,"internal_metadata":1496021549},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,3,4,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:52:31 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.496,"15min":0.53900000000000003},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.69899999999999995,"5min":5.9829999999999997,"15min":15.317},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.626,"15min":0.57499999999999996},"min":{"1min":0.23000000000000001,"5min":0.20100000000000001,"15min":0.184},"max":{"1min":3.734,"5min":6.0750000000000002,"15min":7.5789999999999997},"last":0.52100000000000002}]},{"osd":2,"last update":"Tue Oct 17 01:52:33 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.33800000000000002,"15min":0.34399999999999997},"min":{"1min":0.25800000000000001,"5min":0.20100000000000001,"15min":0.18099999999999999},"max":{"1min":0.49299999999999999,"5min":2.2000000000000002,"15min":3.3580000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.58899999999999997,"15min":0.57399999999999995},"min":{"1min":0.23400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.8550000000000004,"5min":6.1269999999999998,"15min":7.6230000000000002},"last":0.436}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3798253568,"data_stored":4660850688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3682652160,"data_stored":4520484883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7480819712,"data_stored":9181249555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:52:58.432 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:52:58.433 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:52:58.433 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:52:58.434 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:52:58.434 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T01:52:58.795 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:52:58.791+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:53:18.435 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:53:18.895 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:53:18.894+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:53:18.902 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:53:18.898+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:53:19.085 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:53:19.086 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:53:19.144 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2232,"stamp":"2023-10-17T01:53:17.394451+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27161037656,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1861131,"num_read_kb":7248821,"num_write":1834025,"num_write_kb":7335751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83680,"ondisk_log_size":83680,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19925308,"kb_used_data":14668836,"kb_used_omap":1136,"kb_used_meta":5255311,"kb_avail":263190212,"statfs":{"total":289910292480,"available":269506777088,"internally_reserved":0,"allocated":15020888064,"data_stored":18420827131,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1163326,"internal_metadata":5381439426},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,10,7,5,12,18,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18116608,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4549,"num_read_kb":18160,"num_write":4430,"num_write_kb":17720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001993"},"pg_stats":[{"pgid":"2.7","version":"99'12003","reported_seq":17088,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-17T01:53:09.295606+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T01:53:09.295606+0000","last_peered":"2023-10-17T01:53:09.295606+0000","last_clean":"2023-10-17T01:53:09.295606+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:53:09.295606+0000","last_undegraded":"2023-10-17T01:53:09.295606+0000","last_fullsized":"2023-10-17T01:53:09.295606+0000","mapping_epoch":13,"log_start":"56'2000","ondisk_log_start":"56'2000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"99'9043","reported_seq":14027,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-17T01:53:09.274425+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T01:53:09.274425+0000","last_peered":"2023-10-17T01:53:09.274425+0000","last_clean":"2023-10-17T01:53:09.274425+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:53:09.274425+0000","last_undegraded":"2023-10-17T01:53:09.274425+0000","last_fullsized":"2023-10-17T01:53:09.274425+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":9043,"log_dups_size":0,"ondisk_log_size":9043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"99'6063","reported_seq":8685,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-17T01:53:09.250172+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T01:53:09.250172+0000","last_peered":"2023-10-17T01:53:09.250172+0000","last_clean":"2023-10-17T01:53:09.250172+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:53:09.250172+0000","last_undegraded":"2023-10-17T01:53:09.250172+0000","last_fullsized":"2023-10-17T01:53:09.250172+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6063,"log_dups_size":0,"ondisk_log_size":6063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"99'9074","reported_seq":13464,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-17T01:53:09.262310+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T01:53:09.262310+0000","last_peered":"2023-10-17T01:53:09.262310+0000","last_clean":"2023-10-17T01:53:09.262310+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:53:09.262310+0000","last_undegraded":"2023-10-17T01:53:09.262310+0000","last_fullsized":"2023-10-17T01:53:09.262310+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":9074,"log_dups_size":0,"ondisk_log_size":9074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"99'466456","reported_seq":930542,"reported_epoch":99,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:53:17.065591+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T01:53:17.065591+0000","last_peered":"2023-10-17T01:53:17.065591+0000","last_clean":"2023-10-17T01:53:17.065591+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:53:17.065591+0000","last_undegraded":"2023-10-17T01:53:17.065591+0000","last_fullsized":"2023-10-17T01:53:17.065591+0000","mapping_epoch":16,"log_start":"98'456410","ondisk_log_start":"98'456410","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":810,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 303s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":6884810752,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":465100,"num_read_kb":1829088,"num_write":464698,"num_write_kb":1858792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":250,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-17T01:51:21.294145+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:51:21.294145+0000","last_peered":"2023-10-17T01:51:21.294145+0000","last_clean":"2023-10-17T01:51:21.294145+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:51:21.294145+0000","last_undegraded":"2023-10-17T01:51:21.294145+0000","last_fullsized":"2023-10-17T01:51:21.294145+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"99'448850","reported_seq":896209,"reported_epoch":99,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:53:12.553086+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T01:53:12.553086+0000","last_peered":"2023-10-17T01:53:12.553086+0000","last_clean":"2023-10-17T01:53:12.553086+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:53:12.553086+0000","last_undegraded":"2023-10-17T01:53:12.553086+0000","last_fullsized":"2023-10-17T01:53:12.553086+0000","mapping_epoch":16,"log_start":"98'438817","ondisk_log_start":"98'438817","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1000,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 291s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6620213248,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":448311,"num_read_kb":1765020,"num_write":447037,"num_write_kb":1788148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2583,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-17T01:53:12.708825+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T01:53:12.708825+0000","last_peered":"2023-10-17T01:53:12.708825+0000","last_clean":"2023-10-17T01:53:12.708825+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:53:12.708825+0000","last_undegraded":"2023-10-17T01:53:12.708825+0000","last_fullsized":"2023-10-17T01:53:12.708825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"99'468021","reported_seq":933554,"reported_epoch":99,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:53:17.053091+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T01:53:17.053091+0000","last_peered":"2023-10-17T01:53:17.053091+0000","last_clean":"2023-10-17T01:53:17.053091+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:53:17.053091+0000","last_undegraded":"2023-10-17T01:53:17.053091+0000","last_fullsized":"2023-10-17T01:53:17.053091+0000","mapping_epoch":16,"log_start":"98'458018","ondisk_log_start":"98'458018","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":978,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 311s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":6903431168,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":466536,"num_read_kb":1835644,"num_write":466172,"num_write_kb":1864688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":3881,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-17T01:53:12.741459+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T01:53:12.741459+0000","last_peered":"2023-10-17T01:53:12.741459+0000","last_clean":"2023-10-17T01:53:12.741459+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:53:12.741459+0000","last_undegraded":"2023-10-17T01:53:12.741459+0000","last_fullsized":"2023-10-17T01:53:12.741459+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"99'456761","reported_seq":912962,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-17T01:53:12.548376+0000","last_change":"2023-10-17T01:53:10.099706+0000","last_active":"2023-10-17T01:53:12.548376+0000","last_peered":"2023-10-17T01:53:12.548376+0000","last_clean":"2023-10-17T01:53:12.548376+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:53:12.548376+0000","last_undegraded":"2023-10-17T01:53:12.548376+0000","last_fullsized":"2023-10-17T01:53:12.548376+0000","mapping_epoch":16,"log_start":"98'446716","ondisk_log_start":"98'446716","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6751989779,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":457222,"num_read_kb":1798416,"num_write":455488,"num_write_kb":1821946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"99'9043","reported_seq":12796,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-17T01:53:09.320393+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T01:53:09.320393+0000","last_peered":"2023-10-17T01:53:09.320393+0000","last_clean":"2023-10-17T01:53:09.320393+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:53:09.320393+0000","last_undegraded":"2023-10-17T01:53:09.320393+0000","last_fullsized":"2023-10-17T01:53:09.320393+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":9043,"log_dups_size":0,"ondisk_log_size":9043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":271,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-17T01:52:27.120223+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:52:27.120223+0000","last_peered":"2023-10-17T01:52:27.120223+0000","last_clean":"2023-10-17T01:52:27.120223+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:52:27.120223+0000","last_undegraded":"2023-10-17T01:52:27.120223+0000","last_fullsized":"2023-10-17T01:52:27.120223+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27160444947,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1837169,"num_read_kb":7228168,"num_write":1833395,"num_write_kb":7333574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15019012096,"data_stored":18419871782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40127,"ondisk_log_size":40127,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":43454,"ondisk_log_size":43454,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739225,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9084740,"kb_used_data":7333436,"kb_used_omap":562,"kb_used_meta":1750733,"kb_avail":85287100,"statfs":{"total":96636764160,"available":87333990400,"internally_reserved":0,"allocated":7509438464,"data_stored":9209287250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":575753,"internal_metadata":1792751351},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:53:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.45600000000000002,"15min":0.39600000000000002},"min":{"1min":0.19900000000000001,"5min":0.189,"15min":0.16900000000000001},"max":{"1min":3.7679999999999998,"5min":4.2140000000000004,"15min":4.5030000000000001},"last":0.20499999999999999},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.45900000000000002,"15min":0.52100000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":3.5870000000000002,"5min":3.5870000000000002,"15min":5.4240000000000004},"last":0.25800000000000001}]},{"osd":1,"last update":"Tue Oct 17 01:52:30 2023","interfaces":[{"interface":"back","average":{"1min":0.28799999999999998,"5min":0.46800000000000003,"15min":0.48799999999999999},"min":{"1min":0.20699999999999999,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":0.37,"5min":5.7270000000000003,"15min":12.413},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.55600000000000005,"15min":0.51400000000000001},"min":{"1min":0.24399999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":3.0070000000000001,"5min":5.7599999999999998,"15min":5.7599999999999998},"last":0.38700000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739222,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5593844,"kb_used_data":3610924,"kb_used_omap":284,"kb_used_meta":1982627,"kb_avail":88777996,"statfs":{"total":96636764160,"available":90908667904,"internally_reserved":0,"allocated":3697586176,"data_stored":4535208561,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":291170,"internal_metadata":2030210718},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,4,0,5,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:52:53 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.47499999999999998,"15min":0.502},"min":{"1min":0.19400000000000001,"5min":0.17100000000000001,"15min":0.16200000000000001},"max":{"1min":0.86699999999999999,"5min":5.3869999999999996,"15min":23.177},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.52800000000000002,"15min":0.55600000000000005},"min":{"1min":0.246,"5min":0.23100000000000001,"15min":0.19700000000000001},"max":{"1min":0.89400000000000002,"5min":5.4379999999999997,"15min":23.207999999999998},"last":0.49299999999999999}]},{"osd":2,"last update":"Tue Oct 17 01:52:53 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.47099999999999997,"15min":0.55400000000000005},"min":{"1min":0.188,"5min":0.188,"15min":0.184},"max":{"1min":0.64300000000000002,"5min":5.423,"15min":13.757},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.51700000000000002,"15min":0.61799999999999999},"min":{"1min":0.248,"5min":0.24199999999999999,"15min":0.24099999999999999},"max":{"1min":0.68200000000000005,"5min":5.4569999999999999,"15min":14.195},"last":0.42499999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739223,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5246724,"kb_used_data":3724476,"kb_used_omap":289,"kb_used_meta":1521950,"kb_avail":89125116,"statfs":{"total":96636764160,"available":91264118784,"internally_reserved":0,"allocated":3813863424,"data_stored":4676331320,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":296403,"internal_metadata":1558477357},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,0,3,3,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:52:31 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.496,"15min":0.53900000000000003},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.69899999999999995,"5min":5.9829999999999997,"15min":15.317},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.626,"15min":0.57499999999999996},"min":{"1min":0.23000000000000001,"5min":0.20100000000000001,"15min":0.184},"max":{"1min":3.734,"5min":6.0750000000000002,"15min":7.5789999999999997},"last":0.19500000000000001}]},{"osd":2,"last update":"Tue Oct 17 01:52:33 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.33800000000000002,"15min":0.34399999999999997},"min":{"1min":0.25800000000000001,"5min":0.20100000000000001,"15min":0.18099999999999999},"max":{"1min":0.49299999999999999,"5min":2.2000000000000002,"15min":3.3580000000000001},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.58899999999999997,"15min":0.57399999999999995},"min":{"1min":0.23400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.8550000000000004,"5min":6.1269999999999998,"15min":7.6230000000000002},"last":0.378}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3812843520,"data_stored":4675440640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3696742400,"data_stored":4534575123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7509426176,"data_stored":9209856019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:53:19.145 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:53:19.146 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:53:19.146 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:53:19.147 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:53:19.147 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T01:53:21.691 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:53:21.682+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:53:21.695 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:53:21.682+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:53:21.697 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:53:21.690+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:53:21.711 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:53:21.710+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:53:39.148 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:53:39.661 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:53:39.662 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:53:39.695 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2244,"stamp":"2023-10-17T01:53:39.400348+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27193187160,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1869224,"num_read_kb":7281133,"num_write":1841911,"num_write_kb":7367295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84094,"ondisk_log_size":84094,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19165712,"kb_used_data":14728048,"kb_used_omap":1137,"kb_used_meta":4436494,"kb_avail":263949808,"statfs":{"total":289910292480,"available":270284603392,"internally_reserved":0,"allocated":15081521152,"data_stored":18481452191,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1165005,"internal_metadata":4542970163},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,32,8,8,15,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14376960,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3673,"num_read_kb":14680,"num_write":3528,"num_write_kb":14112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001732"},"pg_stats":[{"pgid":"2.7","version":"100'12084","reported_seq":17190,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:53:37.717657+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T01:53:37.717657+0000","last_peered":"2023-10-17T01:53:37.717657+0000","last_clean":"2023-10-17T01:53:37.717657+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:53:37.717657+0000","last_undegraded":"2023-10-17T01:53:37.717657+0000","last_fullsized":"2023-10-17T01:53:37.717657+0000","mapping_epoch":13,"log_start":"56'2000","ondisk_log_start":"56'2000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"100'9103","reported_seq":14103,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:53:37.716611+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T01:53:37.716611+0000","last_peered":"2023-10-17T01:53:37.716611+0000","last_clean":"2023-10-17T01:53:37.716611+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:53:37.716611+0000","last_undegraded":"2023-10-17T01:53:37.716611+0000","last_fullsized":"2023-10-17T01:53:37.716611+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":9103,"log_dups_size":0,"ondisk_log_size":9103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"100'6105","reported_seq":8739,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:53:37.705803+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T01:53:37.705803+0000","last_peered":"2023-10-17T01:53:37.705803+0000","last_clean":"2023-10-17T01:53:37.705803+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:53:37.705803+0000","last_undegraded":"2023-10-17T01:53:37.705803+0000","last_fullsized":"2023-10-17T01:53:37.705803+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6105,"log_dups_size":0,"ondisk_log_size":6105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"100'9136","reported_seq":13543,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:53:37.715794+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T01:53:37.715794+0000","last_peered":"2023-10-17T01:53:37.715794+0000","last_clean":"2023-10-17T01:53:37.715794+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:53:37.715794+0000","last_undegraded":"2023-10-17T01:53:37.715794+0000","last_fullsized":"2023-10-17T01:53:37.715794+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":9136,"log_dups_size":0,"ondisk_log_size":9136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"100'468262","reported_seq":934211,"reported_epoch":100,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:53:37.277516+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T01:53:37.277516+0000","last_peered":"2023-10-17T01:53:37.277516+0000","last_clean":"2023-10-17T01:53:37.277516+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:53:37.277516+0000","last_undegraded":"2023-10-17T01:53:37.277516+0000","last_fullsized":"2023-10-17T01:53:37.277516+0000","mapping_epoch":16,"log_start":"98'458210","ondisk_log_start":"98'458210","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":866,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 324s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":6892023808,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":466947,"num_read_kb":1836456,"num_write":466470,"num_write_kb":1865880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":252,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-17T01:53:27.142168+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:53:27.142168+0000","last_peered":"2023-10-17T01:53:27.142168+0000","last_clean":"2023-10-17T01:53:27.142168+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:53:27.142168+0000","last_undegraded":"2023-10-17T01:53:27.142168+0000","last_fullsized":"2023-10-17T01:53:27.142168+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"100'451052","reported_seq":900576,"reported_epoch":100,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:53:37.774246+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T01:53:37.774246+0000","last_peered":"2023-10-17T01:53:37.774246+0000","last_clean":"2023-10-17T01:53:37.774246+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:53:37.774246+0000","last_undegraded":"2023-10-17T01:53:37.774246+0000","last_fullsized":"2023-10-17T01:53:37.774246+0000","mapping_epoch":16,"log_start":"98'441017","ondisk_log_start":"98'441017","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1092,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 316s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6628958208,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":450465,"num_read_kb":1773624,"num_write":449185,"num_write_kb":1796740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2592,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-17T01:53:33.195681+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T01:53:33.195681+0000","last_peered":"2023-10-17T01:53:33.195681+0000","last_clean":"2023-10-17T01:53:33.195681+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:53:33.195681+0000","last_undegraded":"2023-10-17T01:53:33.195681+0000","last_fullsized":"2023-10-17T01:53:33.195681+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"100'469816","reported_seq":937274,"reported_epoch":100,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:53:37.287852+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T01:53:37.287852+0000","last_peered":"2023-10-17T01:53:37.287852+0000","last_clean":"2023-10-17T01:53:37.287852+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:53:37.287852+0000","last_undegraded":"2023-10-17T01:53:37.287852+0000","last_fullsized":"2023-10-17T01:53:37.287852+0000","mapping_epoch":16,"log_start":"98'459718","ondisk_log_start":"98'459718","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1032,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 332s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":6910644224,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":468446,"num_read_kb":1843276,"num_write":467934,"num_write_kb":1871736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":3893,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-17T01:53:33.195222+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T01:53:33.195222+0000","last_peered":"2023-10-17T01:53:33.195222+0000","last_clean":"2023-10-17T01:53:33.195222+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:53:33.195222+0000","last_undegraded":"2023-10-17T01:53:33.195222+0000","last_fullsized":"2023-10-17T01:53:33.195222+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"100'458965","reported_seq":917361,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:53:37.795516+0000","last_change":"2023-10-17T01:53:37.434659+0000","last_active":"2023-10-17T01:53:37.795516+0000","last_peered":"2023-10-17T01:53:37.795516+0000","last_clean":"2023-10-17T01:53:37.795516+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:53:37.795516+0000","last_undegraded":"2023-10-17T01:53:37.795516+0000","last_fullsized":"2023-10-17T01:53:37.795516+0000","mapping_epoch":16,"log_start":"98'448916","ondisk_log_start":"98'448916","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6760968211,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":459404,"num_read_kb":1807124,"num_write":457692,"num_write_kb":1830762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"100'9105","reported_seq":12875,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:53:37.767751+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T01:53:37.767751+0000","last_peered":"2023-10-17T01:53:37.767751+0000","last_clean":"2023-10-17T01:53:37.767751+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:53:37.767751+0000","last_undegraded":"2023-10-17T01:53:37.767751+0000","last_fullsized":"2023-10-17T01:53:37.767751+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":9105,"log_dups_size":0,"ondisk_log_size":9105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":272,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-17T01:52:37.297610+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:52:37.297610+0000","last_peered":"2023-10-17T01:52:37.297610+0000","last_clean":"2023-10-17T01:52:37.297610+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:52:37.297610+0000","last_undegraded":"2023-10-17T01:52:37.297610+0000","last_fullsized":"2023-10-17T01:52:37.297610+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27192594451,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1845262,"num_read_kb":7260480,"num_write":1841281,"num_write_kb":7365118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15080046592,"data_stored":18480906278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40234,"ondisk_log_size":40234,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":43761,"ondisk_log_size":43761,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739229,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9197144,"kb_used_data":7361168,"kb_used_omap":562,"kb_used_meta":1835405,"kb_avail":85174696,"statfs":{"total":96636764160,"available":87218888704,"internally_reserved":0,"allocated":7537836032,"data_stored":9237684818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":575753,"internal_metadata":1879455479},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:53:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.45600000000000002,"15min":0.39600000000000002},"min":{"1min":0.19900000000000001,"5min":0.189,"15min":0.16900000000000001},"max":{"1min":3.7679999999999998,"5min":4.2140000000000004,"15min":4.5030000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.45900000000000002,"15min":0.52100000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":3.5870000000000002,"5min":3.5870000000000002,"15min":5.4240000000000004},"last":0.32400000000000001}]},{"osd":1,"last update":"Tue Oct 17 01:53:33 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.48899999999999999,"15min":0.51400000000000001},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.19400000000000001},"max":{"1min":5.069,"5min":5.7270000000000003,"15min":12.413},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.52700000000000002,"15min":0.52500000000000002},"min":{"1min":0.248,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":3.4449999999999998,"5min":5.7599999999999998,"15min":5.7599999999999998},"last":0.39100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739227,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4655044,"kb_used_data":3628280,"kb_used_omap":285,"kb_used_meta":1026466,"kb_avail":89716796,"statfs":{"total":96636764160,"available":91869999104,"internally_reserved":0,"allocated":3715358720,"data_stored":4552977091,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":292849,"internal_metadata":1051101199},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,4,10,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:52:53 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.47499999999999998,"15min":0.502},"min":{"1min":0.19400000000000001,"5min":0.17100000000000001,"15min":0.16200000000000001},"max":{"1min":0.86699999999999999,"5min":5.3869999999999996,"15min":23.177},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.52800000000000002,"15min":0.55600000000000005},"min":{"1min":0.246,"5min":0.23100000000000001,"15min":0.19700000000000001},"max":{"1min":0.89400000000000002,"5min":5.4379999999999997,"15min":23.207999999999998},"last":0.80100000000000005}]},{"osd":2,"last update":"Tue Oct 17 01:52:53 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.47099999999999997,"15min":0.55400000000000005},"min":{"1min":0.188,"5min":0.188,"15min":0.184},"max":{"1min":0.64300000000000002,"5min":5.423,"15min":13.757},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.51700000000000002,"15min":0.61799999999999999},"min":{"1min":0.248,"5min":0.24199999999999999,"15min":0.24099999999999999},"max":{"1min":0.68200000000000005,"5min":5.4569999999999999,"15min":14.195},"last":0.44800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739227,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5313524,"kb_used_data":3738600,"kb_used_omap":289,"kb_used_meta":1574622,"kb_avail":89058316,"statfs":{"total":96636764160,"available":91195715584,"internally_reserved":0,"allocated":3828326400,"data_stored":4690790282,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":296403,"internal_metadata":1612413485},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,24,4,0,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:53:33 2023","interfaces":[{"interface":"back","average":{"1min":0.29899999999999999,"5min":0.40200000000000002,"15min":0.53300000000000003},"min":{"1min":0.20100000000000001,"5min":0.17999999999999999,"15min":0.16200000000000001},"max":{"1min":0.69899999999999995,"5min":5.9829999999999997,"15min":15.317},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.629,"15min":0.59699999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.189},"max":{"1min":5.6909999999999998,"5min":6.0750000000000002,"15min":7.5789999999999997},"last":0.25600000000000001}]},{"osd":2,"last update":"Tue Oct 17 01:52:33 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.33800000000000002,"15min":0.34399999999999997},"min":{"1min":0.25800000000000001,"5min":0.20100000000000001,"15min":0.18099999999999999},"max":{"1min":0.49299999999999999,"5min":2.2000000000000002,"15min":3.3580000000000001},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.58899999999999997,"15min":0.57399999999999995},"min":{"1min":0.23400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.8550000000000004,"5min":6.1269999999999998,"15min":7.6230000000000002},"last":0.29199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3827331072,"data_stored":4689928192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3714564096,"data_stored":4552396819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7538151424,"data_stored":9238581267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:53:39.697 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:53:39.697 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:53:39.698 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:53:39.698 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:53:39.698 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T01:53:42.870 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:53:42.866+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:53:42.871 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:53:42.866+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:53:42.871 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:53:42.866+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbda750 2023-10-17T01:53:42.871 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:53:42.866+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:53:42.872 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:53:42.866+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:53:42.872 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:53:42.866+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:53:42.872 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:53:42.866+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:53:42.872 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:53:42.866+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:53:42.873 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:53:42.866+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:53:42.873 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:53:42.866+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:53:42.873 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:53:42.866+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:53:42.874 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:53:42.866+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:53:42.874 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:53:42.866+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:53:42.874 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:53:42.866+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:53:42.874 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:53:42.866+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:53:59.700 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:54:00.232 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:54:00.233 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:54:00.264 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2254,"stamp":"2023-10-17T01:53:59.406542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27222551384,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1876455,"num_read_kb":7309925,"num_write":1849091,"num_write_kb":7396015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84204,"ondisk_log_size":84204,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19372568,"kb_used_data":14785020,"kb_used_omap":1137,"kb_used_meta":4586382,"kb_avail":263742952,"statfs":{"total":289910292480,"available":270072782848,"internally_reserved":0,"allocated":15139860480,"data_stored":18539787505,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1165005,"internal_metadata":4696455475},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,2,6,10,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21942272,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5389,"num_read_kb":21464,"num_write":5364,"num_write_kb":21456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004722"},"pg_stats":[{"pgid":"2.7","version":"100'12148","reported_seq":17266,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:53:53.710817+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T01:53:53.710817+0000","last_peered":"2023-10-17T01:53:53.710817+0000","last_clean":"2023-10-17T01:53:53.710817+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:53:53.710817+0000","last_undegraded":"2023-10-17T01:53:53.710817+0000","last_fullsized":"2023-10-17T01:53:53.710817+0000","mapping_epoch":13,"log_start":"58'2100","ondisk_log_start":"58'2100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"100'9151","reported_seq":14160,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:53:53.693310+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T01:53:53.693310+0000","last_peered":"2023-10-17T01:53:53.693310+0000","last_clean":"2023-10-17T01:53:53.693310+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:53:53.693310+0000","last_undegraded":"2023-10-17T01:53:53.693310+0000","last_fullsized":"2023-10-17T01:53:53.693310+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":9151,"log_dups_size":0,"ondisk_log_size":9151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"100'6137","reported_seq":8778,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:53:53.701476+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T01:53:53.701476+0000","last_peered":"2023-10-17T01:53:53.701476+0000","last_clean":"2023-10-17T01:53:53.701476+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:53:53.701476+0000","last_undegraded":"2023-10-17T01:53:53.701476+0000","last_fullsized":"2023-10-17T01:53:53.701476+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6137,"log_dups_size":0,"ondisk_log_size":6137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"100'9184","reported_seq":13600,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:53:53.709371+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T01:53:53.709371+0000","last_peered":"2023-10-17T01:53:53.709371+0000","last_clean":"2023-10-17T01:53:53.709371+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:53:53.709371+0000","last_undegraded":"2023-10-17T01:53:53.709371+0000","last_fullsized":"2023-10-17T01:53:53.709371+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":9184,"log_dups_size":0,"ondisk_log_size":9184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"100'470151","reported_seq":937973,"reported_epoch":100,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:53:57.482665+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T01:53:57.482665+0000","last_peered":"2023-10-17T01:53:57.482665+0000","last_clean":"2023-10-17T01:53:57.482665+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:53:57.482665+0000","last_undegraded":"2023-10-17T01:53:57.482665+0000","last_fullsized":"2023-10-17T01:53:57.482665+0000","mapping_epoch":16,"log_start":"98'460110","ondisk_log_start":"98'460110","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":912,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 344s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":6899654656,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":468812,"num_read_kb":1843876,"num_write":468336,"num_write_kb":1873344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":252,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-17T01:53:27.142168+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:53:27.142168+0000","last_peered":"2023-10-17T01:53:27.142168+0000","last_clean":"2023-10-17T01:53:27.142168+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:53:27.142168+0000","last_undegraded":"2023-10-17T01:53:27.142168+0000","last_fullsized":"2023-10-17T01:53:27.142168+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"100'452825","reported_seq":904099,"reported_epoch":100,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:53:57.940256+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T01:53:57.940256+0000","last_peered":"2023-10-17T01:53:57.940256+0000","last_clean":"2023-10-17T01:53:57.940256+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:53:57.940256+0000","last_undegraded":"2023-10-17T01:53:57.940256+0000","last_fullsized":"2023-10-17T01:53:57.940256+0000","mapping_epoch":16,"log_start":"98'442817","ondisk_log_start":"98'442817","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1138,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 336s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6636085248,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":452203,"num_read_kb":1780548,"num_write":450929,"num_write_kb":1803716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2601,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:53:53.728314+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T01:53:53.728314+0000","last_peered":"2023-10-17T01:53:53.728314+0000","last_clean":"2023-10-17T01:53:53.728314+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:53:53.728314+0000","last_undegraded":"2023-10-17T01:53:53.728314+0000","last_fullsized":"2023-10-17T01:53:53.728314+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"100'471700","reported_seq":941010,"reported_epoch":100,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:53:57.500162+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T01:53:57.500162+0000","last_peered":"2023-10-17T01:53:57.500162+0000","last_clean":"2023-10-17T01:53:57.500162+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:53:57.500162+0000","last_undegraded":"2023-10-17T01:53:57.500162+0000","last_fullsized":"2023-10-17T01:53:57.500162+0000","mapping_epoch":16,"log_start":"98'461619","ondisk_log_start":"98'461619","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1102,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 352s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":6918193152,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":470292,"num_read_kb":1850628,"num_write":469779,"num_write_kb":1879116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":3906,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:53:53.670909+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T01:53:53.670909+0000","last_peered":"2023-10-17T01:53:53.670909+0000","last_clean":"2023-10-17T01:53:53.670909+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:53:53.670909+0000","last_undegraded":"2023-10-17T01:53:53.670909+0000","last_fullsized":"2023-10-17T01:53:53.670909+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"100'460690","reported_seq":920873,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:53:57.916395+0000","last_change":"2023-10-17T01:53:50.541301+0000","last_active":"2023-10-17T01:53:57.916395+0000","last_peered":"2023-10-17T01:53:57.916395+0000","last_clean":"2023-10-17T01:53:57.916395+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:53:57.916395+0000","last_undegraded":"2023-10-17T01:53:57.916395+0000","last_fullsized":"2023-10-17T01:53:57.916395+0000","mapping_epoch":16,"log_start":"98'450616","ondisk_log_start":"98'450616","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6768025619,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":461186,"num_read_kb":1814220,"num_write":459417,"num_write_kb":1837662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"100'9153","reported_seq":12933,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:53:53.768384+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T01:53:53.768384+0000","last_peered":"2023-10-17T01:53:53.768384+0000","last_clean":"2023-10-17T01:53:53.768384+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:53:53.768384+0000","last_undegraded":"2023-10-17T01:53:53.768384+0000","last_fullsized":"2023-10-17T01:53:53.768384+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":9153,"log_dups_size":0,"ondisk_log_size":9153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":272,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-17T01:52:37.297610+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:52:37.297610+0000","last_peered":"2023-10-17T01:52:37.297610+0000","last_clean":"2023-10-17T01:52:37.297610+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:52:37.297610+0000","last_undegraded":"2023-10-17T01:52:37.297610+0000","last_fullsized":"2023-10-17T01:52:37.297610+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27221958675,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1852493,"num_read_kb":7289272,"num_write":1848461,"num_write_kb":7393838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15138627584,"data_stored":18539487270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40204,"ondisk_log_size":40204,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":43901,"ondisk_log_size":43901,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739233,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9289464,"kb_used_data":7389612,"kb_used_omap":562,"kb_used_meta":1899277,"kb_avail":85082376,"statfs":{"total":96636764160,"available":87124353024,"internally_reserved":0,"allocated":7566962688,"data_stored":9266807460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":575753,"internal_metadata":1944860407},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:53:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.45600000000000002,"15min":0.39600000000000002},"min":{"1min":0.19900000000000001,"5min":0.189,"15min":0.16900000000000001},"max":{"1min":3.7679999999999998,"5min":4.2140000000000004,"15min":4.5030000000000001},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.45900000000000002,"15min":0.52100000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":3.5870000000000002,"5min":3.5870000000000002,"15min":5.4240000000000004},"last":0.24099999999999999}]},{"osd":1,"last update":"Tue Oct 17 01:53:33 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.48899999999999999,"15min":0.51400000000000001},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.19400000000000001},"max":{"1min":5.069,"5min":5.7270000000000003,"15min":12.413},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.52700000000000002,"15min":0.52500000000000002},"min":{"1min":0.248,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":3.4449999999999998,"5min":5.7599999999999998,"15min":5.7599999999999998},"last":0.35999999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739231,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4704944,"kb_used_data":3642152,"kb_used_omap":285,"kb_used_meta":1062498,"kb_avail":89666896,"statfs":{"total":96636764160,"available":91818901504,"internally_reserved":0,"allocated":3729563648,"data_stored":4567182019,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":292849,"internal_metadata":1087997967},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:53:57 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.47299999999999998,"15min":0.501},"min":{"1min":0.19,"5min":0.17100000000000001,"15min":0.16200000000000001},"max":{"1min":0.77300000000000002,"5min":5.3869999999999996,"15min":23.177},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.52700000000000002,"15min":0.55300000000000005},"min":{"1min":0.252,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":0.80100000000000005,"5min":5.4379999999999997,"15min":23.207999999999998},"last":0.30399999999999999}]},{"osd":2,"last update":"Tue Oct 17 01:53:57 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47799999999999998,"15min":0.54400000000000004},"min":{"1min":0.251,"5min":0.188,"15min":0.184},"max":{"1min":0.77400000000000002,"5min":5.423,"15min":13.757},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.52700000000000002,"15min":0.60699999999999998},"min":{"1min":0.33600000000000002,"5min":0.24199999999999999,"15min":0.24099999999999999},"max":{"1min":0.80600000000000005,"5min":5.4569999999999999,"15min":14.195},"last":0.54200000000000004}]}]},{"osd":0,"up_from":8,"seq":34359739231,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5378160,"kb_used_data":3753256,"kb_used_omap":289,"kb_used_meta":1624606,"kb_avail":88993680,"statfs":{"total":96636764160,"available":91129528320,"internally_reserved":0,"allocated":3843334144,"data_stored":4705798026,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":296403,"internal_metadata":1663597101},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,9,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:53:33 2023","interfaces":[{"interface":"back","average":{"1min":0.29899999999999999,"5min":0.40200000000000002,"15min":0.53300000000000003},"min":{"1min":0.20100000000000001,"5min":0.17999999999999999,"15min":0.16200000000000001},"max":{"1min":0.69899999999999995,"5min":5.9829999999999997,"15min":15.317},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.629,"15min":0.59699999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.189},"max":{"1min":5.6909999999999998,"5min":6.0750000000000002,"15min":7.5789999999999997},"last":0.41099999999999998}]},{"osd":2,"last update":"Tue Oct 17 01:53:38 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.34999999999999998,"15min":0.34799999999999998},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.18099999999999999},"max":{"1min":0.89600000000000002,"5min":2.2000000000000002,"15min":3.3580000000000001},"last":0.434},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.65500000000000003,"15min":0.58699999999999997},"min":{"1min":0.222,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":10.378,"5min":10.378,"15min":10.378},"last":0.26500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3842519040,"data_stored":4705116160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3728777216,"data_stored":4566609939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7567331328,"data_stored":9267761171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:54:00.267 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:54:00.267 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:54:00.268 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:54:00.268 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:54:00.268 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T01:54:20.270 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:54:20.810 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:54:20.811 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:54:20.839 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2264,"stamp":"2023-10-17T01:54:19.411096+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27251727192,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1883775,"num_read_kb":7339033,"num_write":1856223,"num_write_kb":7424543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84286,"ondisk_log_size":84286,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19685360,"kb_used_data":14842964,"kb_used_omap":1137,"kb_used_meta":4841230,"kb_avail":263430160,"statfs":{"total":289910292480,"available":269752483840,"internally_reserved":0,"allocated":15199195136,"data_stored":18599122161,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1165005,"internal_metadata":4957419827},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,3,2,6,11,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22110208,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5420,"num_read_kb":21544,"num_write":5403,"num_write_kb":21612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003176"},"pg_stats":[{"pgid":"2.7","version":"100'12188","reported_seq":17321,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:54:14.834067+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T01:54:14.834067+0000","last_peered":"2023-10-17T01:54:14.834067+0000","last_clean":"2023-10-17T01:54:14.834067+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:54:14.834067+0000","last_undegraded":"2023-10-17T01:54:14.834067+0000","last_fullsized":"2023-10-17T01:54:14.834067+0000","mapping_epoch":13,"log_start":"58'2100","ondisk_log_start":"58'2100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"100'9181","reported_seq":14202,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:54:14.822889+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T01:54:14.822889+0000","last_peered":"2023-10-17T01:54:14.822889+0000","last_clean":"2023-10-17T01:54:14.822889+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:54:14.822889+0000","last_undegraded":"2023-10-17T01:54:14.822889+0000","last_fullsized":"2023-10-17T01:54:14.822889+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":9181,"log_dups_size":0,"ondisk_log_size":9181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"100'6157","reported_seq":8805,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:54:14.821253+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T01:54:14.821253+0000","last_peered":"2023-10-17T01:54:14.821253+0000","last_clean":"2023-10-17T01:54:14.821253+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:54:14.821253+0000","last_undegraded":"2023-10-17T01:54:14.821253+0000","last_fullsized":"2023-10-17T01:54:14.821253+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6157,"log_dups_size":0,"ondisk_log_size":6157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"100'9214","reported_seq":13642,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:54:14.823264+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T01:54:14.823264+0000","last_peered":"2023-10-17T01:54:14.823264+0000","last_clean":"2023-10-17T01:54:14.823264+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:54:14.823264+0000","last_undegraded":"2023-10-17T01:54:14.823264+0000","last_fullsized":"2023-10-17T01:54:14.823264+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":9214,"log_dups_size":0,"ondisk_log_size":9214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"100'472016","reported_seq":941719,"reported_epoch":100,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:54:17.604152+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T01:54:17.604152+0000","last_peered":"2023-10-17T01:54:17.604152+0000","last_clean":"2023-10-17T01:54:17.604152+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:54:17.604152+0000","last_undegraded":"2023-10-17T01:54:17.604152+0000","last_fullsized":"2023-10-17T01:54:17.604152+0000","mapping_epoch":16,"log_start":"99'462010","ondisk_log_start":"99'462010","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":960,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 364s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":6907183104,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":470688,"num_read_kb":1851344,"num_write":470176,"num_write_kb":1880704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":252,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-17T01:53:27.142168+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:53:27.142168+0000","last_peered":"2023-10-17T01:53:27.142168+0000","last_clean":"2023-10-17T01:53:27.142168+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:53:27.142168+0000","last_undegraded":"2023-10-17T01:53:27.142168+0000","last_fullsized":"2023-10-17T01:53:27.142168+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"100'454580","reported_seq":907616,"reported_epoch":100,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:54:18.085050+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T01:54:18.085050+0000","last_peered":"2023-10-17T01:54:18.085050+0000","last_clean":"2023-10-17T01:54:18.085050+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:54:18.085050+0000","last_undegraded":"2023-10-17T01:54:18.085050+0000","last_fullsized":"2023-10-17T01:54:18.085050+0000","mapping_epoch":16,"log_start":"98'444517","ondisk_log_start":"98'444517","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1202,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 356s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6643118080,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":453947,"num_read_kb":1787480,"num_write":452649,"num_write_kb":1810596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2609,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:54:14.143617+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T01:54:14.143617+0000","last_peered":"2023-10-17T01:54:14.143617+0000","last_clean":"2023-10-17T01:54:14.143617+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:54:14.143617+0000","last_undegraded":"2023-10-17T01:54:14.143617+0000","last_fullsized":"2023-10-17T01:54:14.143617+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"100'473525","reported_seq":944722,"reported_epoch":100,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:54:17.595889+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T01:54:17.595889+0000","last_peered":"2023-10-17T01:54:17.595889+0000","last_clean":"2023-10-17T01:54:17.595889+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:54:17.595889+0000","last_undegraded":"2023-10-17T01:54:17.595889+0000","last_fullsized":"2023-10-17T01:54:17.595889+0000","mapping_epoch":16,"log_start":"98'463519","ondisk_log_start":"98'463519","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1174,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 372s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":6925475840,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":472165,"num_read_kb":1858072,"num_write":471564,"num_write_kb":1886256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":3918,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:54:14.203512+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T01:54:14.203512+0000","last_peered":"2023-10-17T01:54:14.203512+0000","last_clean":"2023-10-17T01:54:14.203512+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:54:14.203512+0000","last_undegraded":"2023-10-17T01:54:14.203512+0000","last_fullsized":"2023-10-17T01:54:14.203512+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"100'462477","reported_seq":924494,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:54:18.092084+0000","last_change":"2023-10-17T01:54:16.304711+0000","last_active":"2023-10-17T01:54:18.092084+0000","last_peered":"2023-10-17T01:54:18.092084+0000","last_clean":"2023-10-17T01:54:18.092084+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:54:18.092084+0000","last_undegraded":"2023-10-17T01:54:18.092084+0000","last_fullsized":"2023-10-17T01:54:18.092084+0000","mapping_epoch":16,"log_start":"98'452416","ondisk_log_start":"98'452416","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6775357459,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":463013,"num_read_kb":1821484,"num_write":461204,"num_write_kb":1844810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"100'9183","reported_seq":12975,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:54:14.875107+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T01:54:14.875107+0000","last_peered":"2023-10-17T01:54:14.875107+0000","last_clean":"2023-10-17T01:54:14.875107+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:54:14.875107+0000","last_undegraded":"2023-10-17T01:54:14.875107+0000","last_fullsized":"2023-10-17T01:54:14.875107+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":9183,"log_dups_size":0,"ondisk_log_size":9183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":272,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-17T01:52:37.297610+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:52:37.297610+0000","last_peered":"2023-10-17T01:52:37.297610+0000","last_clean":"2023-10-17T01:52:37.297610+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:52:37.297610+0000","last_undegraded":"2023-10-17T01:52:37.297610+0000","last_fullsized":"2023-10-17T01:52:37.297610+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27251134483,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1859813,"num_read_kb":7318380,"num_write":1855593,"num_write_kb":7422366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15197134848,"data_stored":18597994534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40136,"ondisk_log_size":40136,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":44051,"ondisk_log_size":44051,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739237,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9445172,"kb_used_data":7419368,"kb_used_omap":562,"kb_used_meta":2025229,"kb_avail":84926668,"statfs":{"total":96636764160,"available":86964908032,"internally_reserved":0,"allocated":7597432832,"data_stored":9297277604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":575753,"internal_metadata":2073835255},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:54:12 2023","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.42899999999999999,"15min":0.38600000000000001},"min":{"1min":0.20000000000000001,"5min":0.19,"15min":0.16900000000000001},"max":{"1min":0.46000000000000002,"5min":4.2140000000000004,"15min":4.5030000000000001},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.45900000000000002,"15min":0.51300000000000001},"min":{"1min":0.23599999999999999,"5min":0.188,"15min":0.188},"max":{"1min":0.51300000000000001,"5min":3.5870000000000002,"15min":5.4240000000000004},"last":0.27500000000000002}]},{"osd":1,"last update":"Tue Oct 17 01:53:33 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.48899999999999999,"15min":0.51400000000000001},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.19400000000000001},"max":{"1min":5.069,"5min":5.7270000000000003,"15min":12.413},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.52700000000000002,"15min":0.52500000000000002},"min":{"1min":0.248,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":3.4449999999999998,"5min":5.7599999999999998,"15min":5.7599999999999998},"last":0.96699999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739235,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4775568,"kb_used_data":3656008,"kb_used_omap":285,"kb_used_meta":1119266,"kb_avail":89596272,"statfs":{"total":96636764160,"available":91746582528,"internally_reserved":0,"allocated":3743752192,"data_stored":4581370563,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":292849,"internal_metadata":1146128399},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,4,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:53:57 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.47299999999999998,"15min":0.501},"min":{"1min":0.19,"5min":0.17100000000000001,"15min":0.16200000000000001},"max":{"1min":0.77300000000000002,"5min":5.3869999999999996,"15min":23.177},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.52700000000000002,"15min":0.55300000000000005},"min":{"1min":0.252,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":0.80100000000000005,"5min":5.4379999999999997,"15min":23.207999999999998},"last":0.41399999999999998}]},{"osd":2,"last update":"Tue Oct 17 01:53:57 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47799999999999998,"15min":0.54400000000000004},"min":{"1min":0.251,"5min":0.188,"15min":0.184},"max":{"1min":0.77400000000000002,"5min":5.423,"15min":13.757},"last":0.309},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.52700000000000002,"15min":0.60699999999999998},"min":{"1min":0.33600000000000002,"5min":0.24199999999999999,"15min":0.24099999999999999},"max":{"1min":0.80600000000000005,"5min":5.4569999999999999,"15min":14.195},"last":0.34399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739235,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5464620,"kb_used_data":3767588,"kb_used_omap":289,"kb_used_meta":1696734,"kb_avail":88907220,"statfs":{"total":96636764160,"available":91040993280,"internally_reserved":0,"allocated":3858010112,"data_stored":4720473994,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":296403,"internal_metadata":1737456173},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,1,1,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:53:33 2023","interfaces":[{"interface":"back","average":{"1min":0.29899999999999999,"5min":0.40200000000000002,"15min":0.53300000000000003},"min":{"1min":0.20100000000000001,"5min":0.17999999999999999,"15min":0.16200000000000001},"max":{"1min":0.69899999999999995,"5min":5.9829999999999997,"15min":15.317},"last":2.452},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.629,"15min":0.59699999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.189},"max":{"1min":5.6909999999999998,"5min":6.0750000000000002,"15min":7.5789999999999997},"last":0.377}]},{"osd":2,"last update":"Tue Oct 17 01:53:38 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.34999999999999998,"15min":0.34799999999999998},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.18099999999999999},"max":{"1min":0.89600000000000002,"5min":2.2000000000000002,"15min":3.3580000000000001},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.65500000000000003,"15min":0.58699999999999997},"min":{"1min":0.222,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":10.378,"5min":10.378,"15min":10.378},"last":0.40500000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3857371136,"data_stored":4719968256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3743150080,"data_stored":4580982803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7596613632,"data_stored":9297043475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:54:20.841 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:54:20.842 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:54:20.842 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:54:20.843 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:54:20.843 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T01:54:34.177 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:54:34.168+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:54:34.177 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:54:34.172+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:54:34.189 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:54:34.184+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:54:34.198 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:54:34.192+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:54:34.199 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:54:34.192+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:54:34.199 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:54:34.192+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:54:34.199 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:54:34.192+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:54:34.201 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:54:34.196+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:54:34.201 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:54:34.196+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:54:34.201 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:54:34.196+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:54:34.202 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:54:34.196+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:54:34.202 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:54:34.196+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:54:34.202 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:54:34.196+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:54:34.203 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:54:34.196+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:54:40.845 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:54:41.715 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:54:41.716 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:54:41.760 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2275,"stamp":"2023-10-17T01:54:41.414791+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27280837464,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1891047,"num_read_kb":7367901,"num_write":1863337,"num_write_kb":7452999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84436,"ondisk_log_size":84436,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19091360,"kb_used_data":14905800,"kb_used_omap":1244,"kb_used_meta":4184291,"kb_avail":264024160,"statfs":{"total":289910292480,"available":270360739840,"internally_reserved":0,"allocated":15263539200,"data_stored":18663466225,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1274052,"internal_metadata":4284714812},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,3,14,8,10,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14622720,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3659,"num_read_kb":14524,"num_write":3572,"num_write_kb":14288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001991"},"pg_stats":[{"pgid":"2.7","version":"100'12256","reported_seq":17405,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:54:36.065059+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T01:54:36.065059+0000","last_peered":"2023-10-17T01:54:36.065059+0000","last_clean":"2023-10-17T01:54:36.065059+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:54:36.065059+0000","last_undegraded":"2023-10-17T01:54:36.065059+0000","last_fullsized":"2023-10-17T01:54:36.065059+0000","mapping_epoch":13,"log_start":"58'2200","ondisk_log_start":"58'2200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"100'9232","reported_seq":14265,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:54:36.066827+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T01:54:36.066827+0000","last_peered":"2023-10-17T01:54:36.066827+0000","last_clean":"2023-10-17T01:54:36.066827+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:54:36.066827+0000","last_undegraded":"2023-10-17T01:54:36.066827+0000","last_fullsized":"2023-10-17T01:54:36.066827+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":9232,"log_dups_size":0,"ondisk_log_size":9232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"100'6191","reported_seq":8847,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:54:36.052332+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T01:54:36.052332+0000","last_peered":"2023-10-17T01:54:36.052332+0000","last_clean":"2023-10-17T01:54:36.052332+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:54:36.052332+0000","last_undegraded":"2023-10-17T01:54:36.052332+0000","last_fullsized":"2023-10-17T01:54:36.052332+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6191,"log_dups_size":0,"ondisk_log_size":6191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"100'9265","reported_seq":13705,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:54:36.067831+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T01:54:36.067831+0000","last_peered":"2023-10-17T01:54:36.067831+0000","last_clean":"2023-10-17T01:54:36.067831+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:54:36.067831+0000","last_undegraded":"2023-10-17T01:54:36.067831+0000","last_fullsized":"2023-10-17T01:54:36.067831+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":9265,"log_dups_size":0,"ondisk_log_size":9265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"100'473835","reported_seq":945445,"reported_epoch":100,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:54:37.780282+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T01:54:37.780282+0000","last_peered":"2023-10-17T01:54:37.780282+0000","last_clean":"2023-10-17T01:54:37.780282+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:54:37.780282+0000","last_undegraded":"2023-10-17T01:54:37.780282+0000","last_fullsized":"2023-10-17T01:54:37.780282+0000","mapping_epoch":16,"log_start":"99'463810","ondisk_log_start":"99'463810","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1008,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 384s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":6914527232,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":472580,"num_read_kb":1858840,"num_write":471967,"num_write_kb":1887868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":252,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-17T01:53:27.142168+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:53:27.142168+0000","last_peered":"2023-10-17T01:53:27.142168+0000","last_clean":"2023-10-17T01:53:27.142168+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:53:27.142168+0000","last_undegraded":"2023-10-17T01:53:27.142168+0000","last_fullsized":"2023-10-17T01:53:27.142168+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"100'456339","reported_seq":911119,"reported_epoch":100,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:54:38.167115+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T01:54:38.167115+0000","last_peered":"2023-10-17T01:54:38.167115+0000","last_clean":"2023-10-17T01:54:38.167115+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:54:38.167115+0000","last_undegraded":"2023-10-17T01:54:38.167115+0000","last_fullsized":"2023-10-17T01:54:38.167115+0000","mapping_epoch":16,"log_start":"99'446317","ondisk_log_start":"99'446317","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1252,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 376s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6650179584,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":455681,"num_read_kb":1794364,"num_write":454380,"num_write_kb":1817520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2619,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:54:39.757273+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T01:54:39.757273+0000","last_peered":"2023-10-17T01:54:39.757273+0000","last_clean":"2023-10-17T01:54:39.757273+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:54:39.757273+0000","last_undegraded":"2023-10-17T01:54:39.757273+0000","last_fullsized":"2023-10-17T01:54:39.757273+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"100'475366","reported_seq":948461,"reported_epoch":100,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:54:37.790898+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T01:54:37.790898+0000","last_peered":"2023-10-17T01:54:37.790898+0000","last_clean":"2023-10-17T01:54:37.790898+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:54:37.790898+0000","last_undegraded":"2023-10-17T01:54:37.790898+0000","last_fullsized":"2023-10-17T01:54:37.790898+0000","mapping_epoch":16,"log_start":"99'465319","ondisk_log_start":"99'465319","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1220,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 392s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":6932930560,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":474057,"num_read_kb":1865588,"num_write":473380,"num_write_kb":1893520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":3934,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:54:39.767044+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T01:54:39.767044+0000","last_peered":"2023-10-17T01:54:39.767044+0000","last_clean":"2023-10-17T01:54:39.767044+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:54:39.767044+0000","last_undegraded":"2023-10-17T01:54:39.767044+0000","last_fullsized":"2023-10-17T01:54:39.767044+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"100'464253","reported_seq":928027,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:54:38.189233+0000","last_change":"2023-10-17T01:54:32.508799+0000","last_active":"2023-10-17T01:54:38.189233+0000","last_peered":"2023-10-17T01:54:38.189233+0000","last_clean":"2023-10-17T01:54:38.189233+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:54:38.189233+0000","last_undegraded":"2023-10-17T01:54:38.189233+0000","last_fullsized":"2023-10-17T01:54:38.189233+0000","mapping_epoch":16,"log_start":"99'454216","ondisk_log_start":"99'454216","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6782607379,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":464767,"num_read_kb":1828456,"num_write":462980,"num_write_kb":1851914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"100'9234","reported_seq":13038,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:54:36.081038+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T01:54:36.081038+0000","last_peered":"2023-10-17T01:54:36.081038+0000","last_clean":"2023-10-17T01:54:36.081038+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:54:36.081038+0000","last_undegraded":"2023-10-17T01:54:36.081038+0000","last_fullsized":"2023-10-17T01:54:36.081038+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":9234,"log_dups_size":0,"ondisk_log_size":9234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":273,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:53:37.146071+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:53:37.146071+0000","last_peered":"2023-10-17T01:53:37.146071+0000","last_clean":"2023-10-17T01:53:37.146071+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:53:37.146071+0000","last_undegraded":"2023-10-17T01:53:37.146071+0000","last_fullsized":"2023-10-17T01:53:37.146071+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27280244755,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1867085,"num_read_kb":7347248,"num_write":1862707,"num_write_kb":7450822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15262986240,"data_stored":18663845926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40131,"ondisk_log_size":40131,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":44206,"ondisk_log_size":44206,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739242,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8732728,"kb_used_data":7453936,"kb_used_omap":668,"kb_used_meta":1278115,"kb_avail":85639112,"statfs":{"total":96636764160,"available":87694450688,"internally_reserved":0,"allocated":7632830464,"data_stored":9332675236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":684800,"internal_metadata":1308790016},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:54:12 2023","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.42899999999999999,"15min":0.38600000000000001},"min":{"1min":0.20000000000000001,"5min":0.19,"15min":0.16900000000000001},"max":{"1min":0.46000000000000002,"5min":4.2140000000000004,"15min":4.5030000000000001},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.45900000000000002,"15min":0.51300000000000001},"min":{"1min":0.23599999999999999,"5min":0.188,"15min":0.188},"max":{"1min":0.51300000000000001,"5min":3.5870000000000002,"15min":5.4240000000000004},"last":0.33300000000000002}]},{"osd":1,"last update":"Tue Oct 17 01:54:34 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.49199999999999999,"15min":0.51800000000000002},"min":{"1min":0.214,"5min":0.20699999999999999,"15min":0.19400000000000001},"max":{"1min":0.78100000000000003,"5min":5.7270000000000003,"15min":12.413},"last":0.496},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.57099999999999995,"15min":0.53800000000000003},"min":{"1min":0.251,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":2.5920000000000001,"5min":5.7599999999999998,"15min":5.7599999999999998},"last":0.52900000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739239,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4843628,"kb_used_data":3669988,"kb_used_omap":285,"kb_used_meta":1173346,"kb_avail":89528212,"statfs":{"total":96636764160,"available":91676889088,"internally_reserved":0,"allocated":3758067712,"data_stored":4595686083,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":292849,"internal_metadata":1201506319},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,3,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:53:57 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.47299999999999998,"15min":0.501},"min":{"1min":0.19,"5min":0.17100000000000001,"15min":0.16200000000000001},"max":{"1min":0.77300000000000002,"5min":5.3869999999999996,"15min":23.177},"last":0.218},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.52700000000000002,"15min":0.55300000000000005},"min":{"1min":0.252,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":0.80100000000000005,"5min":5.4379999999999997,"15min":23.207999999999998},"last":0.29999999999999999}]},{"osd":2,"last update":"Tue Oct 17 01:53:57 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47799999999999998,"15min":0.54400000000000004},"min":{"1min":0.251,"5min":0.188,"15min":0.184},"max":{"1min":0.77400000000000002,"5min":5.423,"15min":13.757},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.52700000000000002,"15min":0.60699999999999998},"min":{"1min":0.33600000000000002,"5min":0.24199999999999999,"15min":0.24099999999999999},"max":{"1min":0.80600000000000005,"5min":5.4569999999999999,"15min":14.195},"last":0.27500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739239,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5515004,"kb_used_data":3781876,"kb_used_omap":289,"kb_used_meta":1732830,"kb_avail":88856836,"statfs":{"total":96636764160,"available":90989400064,"internally_reserved":0,"allocated":3872641024,"data_stored":4735104906,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":296403,"internal_metadata":1774418477},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,7,1,4,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:54:35 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.39800000000000002,"15min":0.50600000000000001},"min":{"1min":0.19400000000000001,"5min":0.17999999999999999,"15min":0.16200000000000001},"max":{"1min":4.1680000000000001,"5min":4.1680000000000001,"15min":5.9829999999999997},"last":0.253},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.58599999999999997,"15min":0.57299999999999995},"min":{"1min":0.191,"5min":0.191,"15min":0.189},"max":{"1min":2.8450000000000002,"5min":5.6909999999999998,"15min":7.5789999999999997},"last":0.34200000000000003}]},{"osd":2,"last update":"Tue Oct 17 01:53:38 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.34999999999999998,"15min":0.34799999999999998},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.18099999999999999},"max":{"1min":0.89600000000000002,"5min":2.2000000000000002,"15min":3.3580000000000001},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.65500000000000003,"15min":0.58699999999999997},"min":{"1min":0.222,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":10.378,"5min":10.378,"15min":10.378},"last":0.31}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3872141312,"data_stored":4734738432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3757518848,"data_stored":4595351571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7633326080,"data_stored":9333755923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:54:41.762 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:54:41.762 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:54:41.763 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:54:41.763 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:54:41.764 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-17T01:54:42.286 INFO:teuthology.orchestra.run.smithi169.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-17T01:54:42.316 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-17T01:54:42.902 INFO:teuthology.orchestra.run.smithi169.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-17T01:54:42.940 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-17T01:54:43.448 INFO:teuthology.orchestra.run.smithi169.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-17T01:54:43.499 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-17T01:54:44.095 INFO:teuthology.orchestra.run.smithi169.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-17T01:54:44.134 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T01:54:53.751 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:54:53.748+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:54:53.765 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:54:53.760+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:54:53.766 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:54:53.760+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:54:53.784 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:54:53.776+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:55:04.136 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:55:04.614 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:55:04.615 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:55:04.665 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2287,"stamp":"2023-10-17T01:55:03.418433+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27317345112,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1900221,"num_read_kb":7404381,"num_write":1872299,"num_write_kb":7488847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84628,"ondisk_log_size":84628,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19333708,"kb_used_data":14970356,"kb_used_omap":1244,"kb_used_meta":4362083,"kb_avail":263781812,"statfs":{"total":289910292480,"available":270112575488,"internally_reserved":0,"allocated":15329644544,"data_stored":18729559398,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1274052,"internal_metadata":4466773820},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,13,15,3,14],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22073344,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5542,"num_read_kb":22036,"num_write":5427,"num_write_kb":21708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001984"},"pg_stats":[{"pgid":"2.7","version":"101'12313","reported_seq":17483,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-17T01:55:02.636800+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T01:55:02.636800+0000","last_peered":"2023-10-17T01:55:02.636800+0000","last_clean":"2023-10-17T01:55:02.636800+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:55:02.636800+0000","last_undegraded":"2023-10-17T01:55:02.636800+0000","last_fullsized":"2023-10-17T01:55:02.636800+0000","mapping_epoch":13,"log_start":"60'2300","ondisk_log_start":"60'2300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"101'9275","reported_seq":14324,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-17T01:55:02.631635+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T01:55:02.631635+0000","last_peered":"2023-10-17T01:55:02.631635+0000","last_clean":"2023-10-17T01:55:02.631635+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:55:02.631635+0000","last_undegraded":"2023-10-17T01:55:02.631635+0000","last_fullsized":"2023-10-17T01:55:02.631635+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":9275,"log_dups_size":0,"ondisk_log_size":9275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"101'6220","reported_seq":8887,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-17T01:55:02.629510+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T01:55:02.629510+0000","last_peered":"2023-10-17T01:55:02.629510+0000","last_clean":"2023-10-17T01:55:02.629510+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:55:02.629510+0000","last_undegraded":"2023-10-17T01:55:02.629510+0000","last_fullsized":"2023-10-17T01:55:02.629510+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6220,"log_dups_size":0,"ondisk_log_size":6220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"101'9307","reported_seq":13763,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-17T01:55:02.629478+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T01:55:02.629478+0000","last_peered":"2023-10-17T01:55:02.629478+0000","last_clean":"2023-10-17T01:55:02.629478+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:55:02.629478+0000","last_undegraded":"2023-10-17T01:55:02.629478+0000","last_fullsized":"2023-10-17T01:55:02.629478+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":9307,"log_dups_size":0,"ondisk_log_size":9307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"101'476183","reported_seq":950170,"reported_epoch":101,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:55:03.019745+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T01:55:03.019745+0000","last_peered":"2023-10-17T01:55:03.019745+0000","last_clean":"2023-10-17T01:55:03.019745+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:55:03.019745+0000","last_undegraded":"2023-10-17T01:55:03.019745+0000","last_fullsized":"2023-10-17T01:55:03.019745+0000","mapping_epoch":16,"log_start":"99'466110","ondisk_log_start":"99'466110","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1060,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 409s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":6923948032,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":474948,"num_read_kb":1868264,"num_write":474284,"num_write_kb":1897136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":253,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:53:37.140712+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:53:37.140712+0000","last_peered":"2023-10-17T01:53:37.140712+0000","last_clean":"2023-10-17T01:53:37.140712+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:53:37.140712+0000","last_undegraded":"2023-10-17T01:53:37.140712+0000","last_fullsized":"2023-10-17T01:53:37.140712+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"101'458581","reported_seq":915566,"reported_epoch":101,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:55:03.331780+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T01:55:03.331780+0000","last_peered":"2023-10-17T01:55:03.331780+0000","last_clean":"2023-10-17T01:55:03.331780+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:55:03.331780+0000","last_undegraded":"2023-10-17T01:55:03.331780+0000","last_fullsized":"2023-10-17T01:55:03.331780+0000","mapping_epoch":16,"log_start":"99'448518","ondisk_log_start":"99'448518","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1356,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 401s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6659059712,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":457869,"num_read_kb":1803064,"num_write":456561,"num_write_kb":1826244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2628,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-17T01:55:00.244318+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T01:55:00.244318+0000","last_peered":"2023-10-17T01:55:00.244318+0000","last_clean":"2023-10-17T01:55:00.244318+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:55:00.244318+0000","last_undegraded":"2023-10-17T01:55:00.244318+0000","last_fullsized":"2023-10-17T01:55:00.244318+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"101'477654","reported_seq":953098,"reported_epoch":101,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:55:03.004480+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T01:55:03.004480+0000","last_peered":"2023-10-17T01:55:03.004480+0000","last_clean":"2023-10-17T01:55:03.004480+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:55:03.004480+0000","last_undegraded":"2023-10-17T01:55:03.004480+0000","last_fullsized":"2023-10-17T01:55:03.004480+0000","mapping_epoch":16,"log_start":"99'467619","ondisk_log_start":"99'467619","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1268,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 417s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":6942183424,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":476393,"num_read_kb":1874868,"num_write":475643,"num_write_kb":1902572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":3947,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-17T01:55:00.226755+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T01:55:00.226755+0000","last_peered":"2023-10-17T01:55:00.226755+0000","last_clean":"2023-10-17T01:55:00.226755+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:55:00.226755+0000","last_undegraded":"2023-10-17T01:55:00.226755+0000","last_fullsized":"2023-10-17T01:55:00.226755+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"101'466454","reported_seq":932522,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-17T01:55:03.347950+0000","last_change":"2023-10-17T01:54:59.542783+0000","last_active":"2023-10-17T01:55:03.347950+0000","last_peered":"2023-10-17T01:55:03.347950+0000","last_clean":"2023-10-17T01:55:03.347950+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:55:03.347950+0000","last_undegraded":"2023-10-17T01:55:03.347950+0000","last_fullsized":"2023-10-17T01:55:03.347950+0000","mapping_epoch":16,"log_start":"99'456416","ondisk_log_start":"99'456416","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6791561235,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":467049,"num_read_kb":1837532,"num_write":465181,"num_write_kb":1860718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"101'9277","reported_seq":13097,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-17T01:55:02.660091+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T01:55:02.660091+0000","last_peered":"2023-10-17T01:55:02.660091+0000","last_clean":"2023-10-17T01:55:02.660091+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:55:02.660091+0000","last_undegraded":"2023-10-17T01:55:02.660091+0000","last_fullsized":"2023-10-17T01:55:02.660091+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":9277,"log_dups_size":0,"ondisk_log_size":9277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":274,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:54:38.194459+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:54:38.194459+0000","last_peered":"2023-10-17T01:54:38.194459+0000","last_clean":"2023-10-17T01:54:38.194459+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:54:38.194459+0000","last_undegraded":"2023-10-17T01:54:38.194459+0000","last_fullsized":"2023-10-17T01:54:38.194459+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27316752403,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1876259,"num_read_kb":7383728,"num_write":1871669,"num_write_kb":7486670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15328956416,"data_stored":18729816102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40209,"ondisk_log_size":40209,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":44320,"ondisk_log_size":44320,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739246,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8841952,"kb_used_data":7482648,"kb_used_omap":668,"kb_used_meta":1358627,"kb_avail":85529888,"statfs":{"total":96636764160,"available":87582605312,"internally_reserved":0,"allocated":7662231552,"data_stored":9362072267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":684800,"internal_metadata":1391234304},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:54:12 2023","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.42899999999999999,"15min":0.38600000000000001},"min":{"1min":0.20000000000000001,"5min":0.19,"15min":0.16900000000000001},"max":{"1min":0.46000000000000002,"5min":4.2140000000000004,"15min":4.5030000000000001},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.45900000000000002,"15min":0.51300000000000001},"min":{"1min":0.23599999999999999,"5min":0.188,"15min":0.188},"max":{"1min":0.51300000000000001,"5min":3.5870000000000002,"15min":5.4240000000000004},"last":0.32200000000000001}]},{"osd":1,"last update":"Tue Oct 17 01:54:34 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.49199999999999999,"15min":0.51800000000000002},"min":{"1min":0.214,"5min":0.20699999999999999,"15min":0.19400000000000001},"max":{"1min":0.78100000000000003,"5min":5.7270000000000003,"15min":12.413},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.57099999999999995,"15min":0.53800000000000003},"min":{"1min":0.251,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":2.5920000000000001,"5min":5.7599999999999998,"15min":5.7599999999999998},"last":0.37}]}]},{"osd":1,"up_from":8,"seq":34359739244,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4897292,"kb_used_data":3687556,"kb_used_omap":285,"kb_used_meta":1209442,"kb_avail":89474548,"statfs":{"total":96636764160,"available":91621937152,"internally_reserved":0,"allocated":3776057344,"data_stored":4613671658,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":292849,"internal_metadata":1238468623},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,5,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:54:58 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.57399999999999995,"15min":0.52400000000000002},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.16200000000000001},"max":{"1min":6.6890000000000001,"5min":6.6890000000000001,"15min":23.177},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.627,"15min":0.57499999999999996},"min":{"1min":0.25800000000000001,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":6.7539999999999996,"5min":6.7539999999999996,"15min":23.207999999999998},"last":0.313}]},{"osd":2,"last update":"Tue Oct 17 01:54:58 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.48199999999999998,"15min":0.53300000000000003},"min":{"1min":0.192,"5min":0.188,"15min":0.184},"max":{"1min":0.83899999999999997,"5min":5.423,"15min":13.757},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.376,"5min":0.52500000000000002,"15min":0.59499999999999997},"min":{"1min":0.255,"5min":0.24199999999999999,"15min":0.24099999999999999},"max":{"1min":0.86899999999999999,"5min":5.4569999999999999,"15min":14.195},"last":0.29799999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739244,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5594464,"kb_used_data":3800152,"kb_used_omap":289,"kb_used_meta":1794014,"kb_avail":88777376,"statfs":{"total":96636764160,"available":90908033024,"internally_reserved":0,"allocated":3891355648,"data_stored":4753815473,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":296403,"internal_metadata":1837070893},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,6,8,1,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:54:35 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.39800000000000002,"15min":0.50600000000000001},"min":{"1min":0.19400000000000001,"5min":0.17999999999999999,"15min":0.16200000000000001},"max":{"1min":4.1680000000000001,"5min":4.1680000000000001,"15min":5.9829999999999997},"last":0.249},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.58599999999999997,"15min":0.57299999999999995},"min":{"1min":0.191,"5min":0.191,"15min":0.189},"max":{"1min":2.8450000000000002,"5min":5.6909999999999998,"15min":7.5789999999999997},"last":0.41199999999999998}]},{"osd":2,"last update":"Tue Oct 17 01:54:41 2023","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.36399999999999999,"15min":0.36099999999999999},"min":{"1min":0.191,"5min":0.191,"15min":0.18099999999999999},"max":{"1min":2.6120000000000001,"5min":2.6120000000000001,"15min":3.3580000000000001},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.68799999999999994,"15min":0.58499999999999996},"min":{"1min":0.23999999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.46,"5min":10.378,"15min":10.378},"last":0.45100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3890913280,"data_stored":4753510400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3775463424,"data_stored":4613296147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7662579712,"data_stored":9363009555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:55:04.670 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:55:04.670 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:55:04.671 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:55:04.671 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:55:04.671 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T01:55:05.182 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:55:05.176+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:55:05.192 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:55:05.184+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:55:05.193 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:55:05.184+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbda750 2023-10-17T01:55:05.193 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:55:05.188+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:55:05.193 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:55:05.188+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:55:05.194 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:55:05.188+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:55:05.194 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:55:05.188+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:55:05.194 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:55:05.188+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:55:05.195 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:55:05.188+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:55:05.195 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:55:05.188+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:55:05.195 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:55:05.188+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:55:05.195 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:55:05.188+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:55:05.196 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:55:05.188+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:55:05.196 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:55:05.188+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:55:05.196 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:55:05.188+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:55:24.672 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:55:25.271 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:55:25.272 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:55:25.307 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2297,"stamp":"2023-10-17T01:55:23.422011+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27343768408,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1906403,"num_read_kb":7428973,"num_write":1878770,"num_write_kb":7514731,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84676,"ondisk_log_size":84676,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19561232,"kb_used_data":15025780,"kb_used_omap":1244,"kb_used_meta":4534179,"kb_avail":263554288,"statfs":{"total":289910292480,"available":269879590912,"internally_reserved":0,"allocated":15386398720,"data_stored":18786313574,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1274052,"internal_metadata":4643000124},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,3,4,5,10,17,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18984960,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4459,"num_read_kb":17752,"num_write":4646,"num_write_kb":18584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002122"},"pg_stats":[{"pgid":"2.7","version":"101'12345","reported_seq":17527,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-17T01:55:18.285121+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T01:55:18.285121+0000","last_peered":"2023-10-17T01:55:18.285121+0000","last_clean":"2023-10-17T01:55:18.285121+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:55:18.285121+0000","last_undegraded":"2023-10-17T01:55:18.285121+0000","last_fullsized":"2023-10-17T01:55:18.285121+0000","mapping_epoch":13,"log_start":"60'2300","ondisk_log_start":"60'2300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"101'9299","reported_seq":14357,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-17T01:55:18.287577+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T01:55:18.287577+0000","last_peered":"2023-10-17T01:55:18.287577+0000","last_clean":"2023-10-17T01:55:18.287577+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:55:18.287577+0000","last_undegraded":"2023-10-17T01:55:18.287577+0000","last_fullsized":"2023-10-17T01:55:18.287577+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":9299,"log_dups_size":0,"ondisk_log_size":9299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"101'6236","reported_seq":8909,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-17T01:55:18.278856+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T01:55:18.278856+0000","last_peered":"2023-10-17T01:55:18.278856+0000","last_clean":"2023-10-17T01:55:18.278856+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:55:18.278856+0000","last_undegraded":"2023-10-17T01:55:18.278856+0000","last_fullsized":"2023-10-17T01:55:18.278856+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6236,"log_dups_size":0,"ondisk_log_size":6236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"101'9331","reported_seq":13796,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-17T01:55:18.291185+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T01:55:18.291185+0000","last_peered":"2023-10-17T01:55:18.291185+0000","last_clean":"2023-10-17T01:55:18.291185+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:55:18.291185+0000","last_undegraded":"2023-10-17T01:55:18.291185+0000","last_fullsized":"2023-10-17T01:55:18.291185+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":9331,"log_dups_size":0,"ondisk_log_size":9331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"101'478128","reported_seq":953974,"reported_epoch":101,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:55:23.187559+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T01:55:23.187559+0000","last_peered":"2023-10-17T01:55:23.187559+0000","last_clean":"2023-10-17T01:55:23.187559+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:55:23.187559+0000","last_undegraded":"2023-10-17T01:55:23.187559+0000","last_fullsized":"2023-10-17T01:55:23.187559+0000","mapping_epoch":16,"log_start":"99'468110","ondisk_log_start":"99'468110","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1080,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 430s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":6931841024,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":476798,"num_read_kb":1875648,"num_write":476218,"num_write_kb":1904872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":253,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:53:37.140712+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:53:37.140712+0000","last_peered":"2023-10-17T01:53:37.140712+0000","last_clean":"2023-10-17T01:53:37.140712+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:53:37.140712+0000","last_undegraded":"2023-10-17T01:53:37.140712+0000","last_fullsized":"2023-10-17T01:53:37.140712+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"101'459953","reported_seq":918186,"reported_epoch":101,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:55:18.448528+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T01:55:18.448528+0000","last_peered":"2023-10-17T01:55:18.448528+0000","last_clean":"2023-10-17T01:55:18.448528+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:55:18.448528+0000","last_undegraded":"2023-10-17T01:55:18.448528+0000","last_fullsized":"2023-10-17T01:55:18.448528+0000","mapping_epoch":16,"log_start":"99'449918","ondisk_log_start":"99'449918","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1380,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 416s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6664593408,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":459110,"num_read_kb":1807988,"num_write":457919,"num_write_kb":1831676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2636,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-17T01:55:20.704487+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T01:55:20.704487+0000","last_peered":"2023-10-17T01:55:20.704487+0000","last_clean":"2023-10-17T01:55:20.704487+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:55:20.704487+0000","last_undegraded":"2023-10-17T01:55:20.704487+0000","last_fullsized":"2023-10-17T01:55:20.704487+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"101'479536","reported_seq":956772,"reported_epoch":101,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:55:23.153393+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T01:55:23.153393+0000","last_peered":"2023-10-17T01:55:23.153393+0000","last_clean":"2023-10-17T01:55:23.153393+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:55:23.153393+0000","last_undegraded":"2023-10-17T01:55:23.153393+0000","last_fullsized":"2023-10-17T01:55:23.153393+0000","mapping_epoch":16,"log_start":"99'469519","ondisk_log_start":"99'469519","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1332,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 438s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":6949748736,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":478170,"num_read_kb":1881916,"num_write":477493,"num_write_kb":1909972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":3959,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-17T01:55:20.703927+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T01:55:20.703927+0000","last_peered":"2023-10-17T01:55:20.703927+0000","last_clean":"2023-10-17T01:55:20.703927+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:55:20.703927+0000","last_undegraded":"2023-10-17T01:55:20.703927+0000","last_fullsized":"2023-10-17T01:55:20.703927+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"101'467783","reported_seq":935167,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-17T01:55:18.408362+0000","last_change":"2023-10-17T01:55:11.478450+0000","last_active":"2023-10-17T01:55:18.408362+0000","last_peered":"2023-10-17T01:55:18.408362+0000","last_clean":"2023-10-17T01:55:18.408362+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:55:18.408362+0000","last_undegraded":"2023-10-17T01:55:18.408362+0000","last_fullsized":"2023-10-17T01:55:18.408362+0000","mapping_epoch":16,"log_start":"99'457716","ondisk_log_start":"99'457716","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6796992531,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":468363,"num_read_kb":1842768,"num_write":466510,"num_write_kb":1866034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"101'9301","reported_seq":13130,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-17T01:55:18.326902+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T01:55:18.326902+0000","last_peered":"2023-10-17T01:55:18.326902+0000","last_clean":"2023-10-17T01:55:18.326902+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:55:18.326902+0000","last_undegraded":"2023-10-17T01:55:18.326902+0000","last_fullsized":"2023-10-17T01:55:18.326902+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":9301,"log_dups_size":0,"ondisk_log_size":9301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":274,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:54:38.194459+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:54:38.194459+0000","last_peered":"2023-10-17T01:54:38.194459+0000","last_clean":"2023-10-17T01:54:38.194459+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:54:38.194459+0000","last_undegraded":"2023-10-17T01:54:38.194459+0000","last_fullsized":"2023-10-17T01:54:38.194459+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27343175699,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1882441,"num_read_kb":7408320,"num_write":1878140,"num_write_kb":7512554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15385616384,"data_stored":18786476070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40137,"ondisk_log_size":40137,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":44440,"ondisk_log_size":44440,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739250,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8957520,"kb_used_data":7512132,"kb_used_omap":668,"kb_used_meta":1444707,"kb_avail":85414320,"statfs":{"total":96636764160,"available":87464263680,"internally_reserved":0,"allocated":7692423168,"data_stored":9392263883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":684800,"internal_metadata":1479380224},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:55:12 2023","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.45400000000000001,"15min":0.39300000000000002},"min":{"1min":0.189,"5min":0.189,"15min":0.16900000000000001},"max":{"1min":2.5790000000000002,"5min":4.2140000000000004,"15min":4.5030000000000001},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.497,"15min":0.502},"min":{"1min":0.19800000000000001,"5min":0.188,"15min":0.188},"max":{"1min":2.5419999999999998,"5min":3.5870000000000002,"15min":5.4240000000000004},"last":0.51600000000000001}]},{"osd":1,"last update":"Tue Oct 17 01:54:34 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.49199999999999999,"15min":0.51800000000000002},"min":{"1min":0.214,"5min":0.20699999999999999,"15min":0.19400000000000001},"max":{"1min":0.78100000000000003,"5min":5.7270000000000003,"15min":12.413},"last":0.373},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.57099999999999995,"15min":0.53800000000000003},"min":{"1min":0.251,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":2.5920000000000001,"5min":5.7599999999999998,"15min":5.7599999999999998},"last":0.34200000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739247,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4957888,"kb_used_data":3698232,"kb_used_omap":285,"kb_used_meta":1259362,"kb_avail":89413952,"statfs":{"total":96636764160,"available":91559886848,"internally_reserved":0,"allocated":3786989568,"data_stored":4624603882,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":292849,"internal_metadata":1289586703},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,0,6,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:54:58 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.57399999999999995,"15min":0.52400000000000002},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.16200000000000001},"max":{"1min":6.6890000000000001,"5min":6.6890000000000001,"15min":23.177},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.627,"15min":0.57499999999999996},"min":{"1min":0.25800000000000001,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":6.7539999999999996,"5min":6.7539999999999996,"15min":23.207999999999998},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Oct 17 01:54:58 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.48199999999999998,"15min":0.53300000000000003},"min":{"1min":0.192,"5min":0.188,"15min":0.184},"max":{"1min":0.83899999999999997,"5min":5.423,"15min":13.757},"last":0.25},{"interface":"front","average":{"1min":0.376,"5min":0.52500000000000002,"15min":0.59499999999999997},"min":{"1min":0.255,"5min":0.24199999999999999,"15min":0.24099999999999999},"max":{"1min":0.86899999999999999,"5min":5.4569999999999999,"15min":14.195},"last":0.28999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739248,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5645824,"kb_used_data":3815416,"kb_used_omap":289,"kb_used_meta":1830110,"kb_avail":88726016,"statfs":{"total":96636764160,"available":90855440384,"internally_reserved":0,"allocated":3906985984,"data_stored":4769445809,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":296403,"internal_metadata":1874033197},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,0,2,1,8,11,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:54:35 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.39800000000000002,"15min":0.50600000000000001},"min":{"1min":0.19400000000000001,"5min":0.17999999999999999,"15min":0.16200000000000001},"max":{"1min":4.1680000000000001,"5min":4.1680000000000001,"15min":5.9829999999999997},"last":0.187},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.58599999999999997,"15min":0.57299999999999995},"min":{"1min":0.191,"5min":0.191,"15min":0.189},"max":{"1min":2.8450000000000002,"5min":5.6909999999999998,"15min":7.5789999999999997},"last":0.23499999999999999}]},{"osd":2,"last update":"Tue Oct 17 01:54:41 2023","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.36399999999999999,"15min":0.36099999999999999},"min":{"1min":0.191,"5min":0.191,"15min":0.18099999999999999},"max":{"1min":2.6120000000000001,"5min":2.6120000000000001,"15min":3.3580000000000001},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.68799999999999994,"15min":0.58499999999999996},"min":{"1min":0.23999999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.46,"5min":10.378,"15min":10.378},"last":0.30099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3906412544,"data_stored":4769009664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3786473472,"data_stored":4624306195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7692730368,"data_stored":9393160211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:55:25.309 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:55:25.309 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:55:25.310 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:55:25.310 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:55:25.310 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T01:55:26.034 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:55:26.027+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:55:29.239 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:55:29.235+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:55:29.248 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:55:29.243+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:55:45.312 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:55:45.810 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:55:45.811 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:55:45.849 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2308,"stamp":"2023-10-17T01:55:45.425766+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27377249112,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1914139,"num_read_kb":7459725,"num_write":1886954,"num_write_kb":7547467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84956,"ondisk_log_size":84956,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19822824,"kb_used_data":15087628,"kb_used_omap":1244,"kb_used_meta":4733923,"kb_avail":263292696,"statfs":{"total":289910292480,"available":269611720704,"internally_reserved":0,"allocated":15449731072,"data_stored":18849645926,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1274052,"internal_metadata":4847537980},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,8,7,3,11,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18849792,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4350,"num_read_kb":17296,"num_write":4596,"num_write_kb":18384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002018"},"pg_stats":[{"pgid":"2.7","version":"101'12421","reported_seq":17623,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-17T01:55:41.918906+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T01:55:41.918906+0000","last_peered":"2023-10-17T01:55:41.918906+0000","last_clean":"2023-10-17T01:55:41.918906+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:55:41.918906+0000","last_undegraded":"2023-10-17T01:55:41.918906+0000","last_fullsized":"2023-10-17T01:55:41.918906+0000","mapping_epoch":13,"log_start":"60'2400","ondisk_log_start":"60'2400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"101'9356","reported_seq":14429,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-17T01:55:41.898971+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T01:55:41.898971+0000","last_peered":"2023-10-17T01:55:41.898971+0000","last_clean":"2023-10-17T01:55:41.898971+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:55:41.898971+0000","last_undegraded":"2023-10-17T01:55:41.898971+0000","last_fullsized":"2023-10-17T01:55:41.898971+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":9356,"log_dups_size":0,"ondisk_log_size":9356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"101'6275","reported_seq":8959,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-17T01:55:41.874793+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T01:55:41.874793+0000","last_peered":"2023-10-17T01:55:41.874793+0000","last_clean":"2023-10-17T01:55:41.874793+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:55:41.874793+0000","last_undegraded":"2023-10-17T01:55:41.874793+0000","last_fullsized":"2023-10-17T01:55:41.874793+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6275,"log_dups_size":0,"ondisk_log_size":6275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"101'9389","reported_seq":13870,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-17T01:55:41.901677+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T01:55:41.901677+0000","last_peered":"2023-10-17T01:55:41.901677+0000","last_clean":"2023-10-17T01:55:41.901677+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:55:41.901677+0000","last_undegraded":"2023-10-17T01:55:41.901677+0000","last_fullsized":"2023-10-17T01:55:41.901677+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":9389,"log_dups_size":0,"ondisk_log_size":9389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"101'480026","reported_seq":957633,"reported_epoch":101,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:55:43.330677+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T01:55:43.330677+0000","last_peered":"2023-10-17T01:55:43.330677+0000","last_clean":"2023-10-17T01:55:43.330677+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:55:43.330677+0000","last_undegraded":"2023-10-17T01:55:43.330677+0000","last_fullsized":"2023-10-17T01:55:43.330677+0000","mapping_epoch":16,"log_start":"100'470010","ondisk_log_start":"100'470010","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1114,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 450s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":6939521024,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":478551,"num_read_kb":1882624,"num_write":478097,"num_write_kb":1912388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":254,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-17T01:54:42.179415+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:54:42.179415+0000","last_peered":"2023-10-17T01:54:42.179415+0000","last_clean":"2023-10-17T01:54:42.179415+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:54:42.179415+0000","last_undegraded":"2023-10-17T01:54:42.179415+0000","last_fullsized":"2023-10-17T01:54:42.179415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"101'462174","reported_seq":922517,"reported_epoch":101,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:55:43.629471+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T01:55:43.629471+0000","last_peered":"2023-10-17T01:55:43.629471+0000","last_clean":"2023-10-17T01:55:43.629471+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:55:43.629471+0000","last_undegraded":"2023-10-17T01:55:43.629471+0000","last_fullsized":"2023-10-17T01:55:43.629471+0000","mapping_epoch":16,"log_start":"100'452118","ondisk_log_start":"100'452118","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1478,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 442s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6673457152,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":461210,"num_read_kb":1816336,"num_write":460086,"num_write_kb":1840344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2643,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-17T01:55:36.063468+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T01:55:36.063468+0000","last_peered":"2023-10-17T01:55:36.063468+0000","last_clean":"2023-10-17T01:55:36.063468+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:55:36.063468+0000","last_undegraded":"2023-10-17T01:55:36.063468+0000","last_fullsized":"2023-10-17T01:55:36.063468+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"101'481403","reported_seq":960357,"reported_epoch":101,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:55:43.329201+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T01:55:43.329201+0000","last_peered":"2023-10-17T01:55:43.329201+0000","last_clean":"2023-10-17T01:55:43.329201+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:55:43.329201+0000","last_undegraded":"2023-10-17T01:55:43.329201+0000","last_fullsized":"2023-10-17T01:55:43.329201+0000","mapping_epoch":16,"log_start":"100'471319","ondisk_log_start":"100'471319","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1388,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 458s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":6957248512,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":479882,"num_read_kb":1888720,"num_write":479326,"num_write_kb":1917304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":3968,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-17T01:55:36.139735+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T01:55:36.139735+0000","last_peered":"2023-10-17T01:55:36.139735+0000","last_clean":"2023-10-17T01:55:36.139735+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:55:36.139735+0000","last_undegraded":"2023-10-17T01:55:36.139735+0000","last_fullsized":"2023-10-17T01:55:36.139735+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"101'470088","reported_seq":939650,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-17T01:55:43.634465+0000","last_change":"2023-10-17T01:55:42.806215+0000","last_active":"2023-10-17T01:55:43.634465+0000","last_peered":"2023-10-17T01:55:43.634465+0000","last_clean":"2023-10-17T01:55:43.634465+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:55:43.634465+0000","last_undegraded":"2023-10-17T01:55:43.634465+0000","last_fullsized":"2023-10-17T01:55:43.634465+0000","mapping_epoch":16,"log_start":"100'460016","ondisk_log_start":"100'460016","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6806429715,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":470534,"num_read_kb":1851392,"num_write":468815,"num_write_kb":1875254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"101'9360","reported_seq":13205,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-17T01:55:41.887269+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T01:55:41.887269+0000","last_peered":"2023-10-17T01:55:41.887269+0000","last_clean":"2023-10-17T01:55:41.887269+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:55:41.887269+0000","last_undegraded":"2023-10-17T01:55:41.887269+0000","last_fullsized":"2023-10-17T01:55:41.887269+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":9360,"log_dups_size":0,"ondisk_log_size":9360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":274,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-17T01:54:38.194459+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:54:38.194459+0000","last_peered":"2023-10-17T01:54:38.194459+0000","last_clean":"2023-10-17T01:54:38.194459+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:54:38.194459+0000","last_undegraded":"2023-10-17T01:54:38.194459+0000","last_fullsized":"2023-10-17T01:54:38.194459+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27376656403,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1890177,"num_read_kb":7439072,"num_write":1886324,"num_write_kb":7545290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15448940544,"data_stored":18849800230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40228,"ondisk_log_size":40228,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":44629,"ondisk_log_size":44629,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739254,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9085112,"kb_used_data":7541168,"kb_used_omap":668,"kb_used_meta":1543267,"kb_avail":85286728,"statfs":{"total":96636764160,"available":87333609472,"internally_reserved":0,"allocated":7722156032,"data_stored":9421996747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":684800,"internal_metadata":1580305664},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:55:12 2023","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.45400000000000001,"15min":0.39300000000000002},"min":{"1min":0.189,"5min":0.189,"15min":0.16900000000000001},"max":{"1min":2.5790000000000002,"5min":4.2140000000000004,"15min":4.5030000000000001},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.497,"15min":0.502},"min":{"1min":0.19800000000000001,"5min":0.188,"15min":0.188},"max":{"1min":2.5419999999999998,"5min":3.5870000000000002,"15min":5.4240000000000004},"last":0.34499999999999997}]},{"osd":1,"last update":"Tue Oct 17 01:55:35 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.47499999999999998,"15min":0.51500000000000001},"min":{"1min":0.23899999999999999,"5min":0.20699999999999999,"15min":0.19400000000000001},"max":{"1min":0.496,"5min":5.7270000000000003,"15min":12.413},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60599999999999998,"15min":0.55400000000000005},"min":{"1min":0.28999999999999998,"5min":0.24399999999999999,"15min":0.19500000000000001},"max":{"1min":4.8739999999999997,"5min":5.7599999999999998,"15min":5.7599999999999998},"last":0.312}]}]},{"osd":1,"up_from":8,"seq":34359739252,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5040848,"kb_used_data":3716100,"kb_used_omap":285,"kb_used_meta":1324450,"kb_avail":89330992,"statfs":{"total":96636764160,"available":91474935808,"internally_reserved":0,"allocated":3805286400,"data_stored":4642900714,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":292849,"internal_metadata":1356236815},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,6,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:54:58 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.57399999999999995,"15min":0.52400000000000002},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.16200000000000001},"max":{"1min":6.6890000000000001,"5min":6.6890000000000001,"15min":23.177},"last":3.71},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.627,"15min":0.57499999999999996},"min":{"1min":0.25800000000000001,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":6.7539999999999996,"5min":6.7539999999999996,"15min":23.207999999999998},"last":3.7509999999999999}]},{"osd":2,"last update":"Tue Oct 17 01:54:58 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.48199999999999998,"15min":0.53300000000000003},"min":{"1min":0.192,"5min":0.188,"15min":0.184},"max":{"1min":0.83899999999999997,"5min":5.423,"15min":13.757},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.376,"5min":0.52500000000000002,"15min":0.59499999999999997},"min":{"1min":0.255,"5min":0.24199999999999999,"15min":0.24099999999999999},"max":{"1min":0.86899999999999999,"5min":5.4569999999999999,"15min":14.195},"last":0.318}]}]},{"osd":0,"up_from":8,"seq":34359739252,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5696864,"kb_used_data":3830360,"kb_used_omap":289,"kb_used_meta":1866206,"kb_avail":88674976,"statfs":{"total":96636764160,"available":90803175424,"internally_reserved":0,"allocated":3922288640,"data_stored":4784748465,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":296403,"internal_metadata":1910995501},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,7,2,0,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:55:35 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.38700000000000001,"15min":0.499},"min":{"1min":0.186,"5min":0.17999999999999999,"15min":0.16200000000000001},"max":{"1min":0.52100000000000002,"5min":4.1680000000000001,"15min":5.9829999999999997},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.48199999999999998,"15min":0.53300000000000003},"min":{"1min":0.192,"5min":0.191,"15min":0.189},"max":{"1min":0.54900000000000004,"5min":5.6909999999999998,"15min":6.0750000000000002},"last":0.22500000000000001}]},{"osd":2,"last update":"Tue Oct 17 01:54:41 2023","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.36399999999999999,"15min":0.36099999999999999},"min":{"1min":0.191,"5min":0.191,"15min":0.18099999999999999},"max":{"1min":2.6120000000000001,"5min":2.6120000000000001,"15min":3.3580000000000001},"last":0.255},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.68799999999999994,"15min":0.58499999999999996},"min":{"1min":0.23999999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.46,"5min":10.378,"15min":10.378},"last":0.27100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3921604608,"data_stored":4784201728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3804778496,"data_stored":4642611219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7722557440,"data_stored":9422987283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:55:45.852 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:55:45.852 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:55:45.852 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:55:45.853 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:55:45.853 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T01:56:00.624 INFO:tasks.ceph.mgr.x.smithi169.stderr:2023-10-17T01:56:00.618+0000 7f35060d7640 -1 calc_pg_upmaps abort due to max <= 0 2023-10-17T01:56:05.855 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:56:06.384 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:56:06.385 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:56:06.427 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2319,"stamp":"2023-10-17T01:56:05.429295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27406654296,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1920771,"num_read_kb":7486169,"num_write":1894153,"num_write_kb":7576263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85064,"ondisk_log_size":85064,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20121952,"kb_used_data":15145732,"kb_used_omap":1244,"kb_used_meta":4974947,"kb_avail":262993568,"statfs":{"total":289910292480,"available":269305413632,"internally_reserved":0,"allocated":15509229568,"data_stored":18909136222,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1274052,"internal_metadata":5094346556},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,7,2,18,12,15],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14893056,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3290,"num_read_kb":13144,"num_write":3637,"num_write_kb":14548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001748"},"pg_stats":[{"pgid":"2.7","version":"102'12481","reported_seq":17700,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-17T01:56:02.910858+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T01:56:02.910858+0000","last_peered":"2023-10-17T01:56:02.910858+0000","last_clean":"2023-10-17T01:56:02.910858+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:56:02.910858+0000","last_undegraded":"2023-10-17T01:56:02.910858+0000","last_fullsized":"2023-10-17T01:56:02.910858+0000","mapping_epoch":13,"log_start":"60'2400","ondisk_log_start":"60'2400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"102'9401","reported_seq":14487,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-17T01:56:02.906205+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T01:56:02.906205+0000","last_peered":"2023-10-17T01:56:02.906205+0000","last_clean":"2023-10-17T01:56:02.906205+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:56:02.906205+0000","last_undegraded":"2023-10-17T01:56:02.906205+0000","last_fullsized":"2023-10-17T01:56:02.906205+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":9401,"log_dups_size":0,"ondisk_log_size":9401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"102'6305","reported_seq":8998,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-17T01:56:02.884874+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T01:56:02.884874+0000","last_peered":"2023-10-17T01:56:02.884874+0000","last_clean":"2023-10-17T01:56:02.884874+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:56:02.884874+0000","last_undegraded":"2023-10-17T01:56:02.884874+0000","last_fullsized":"2023-10-17T01:56:02.884874+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6305,"log_dups_size":0,"ondisk_log_size":6305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"102'9434","reported_seq":13928,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-17T01:56:02.903967+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T01:56:02.903967+0000","last_peered":"2023-10-17T01:56:02.903967+0000","last_clean":"2023-10-17T01:56:02.903967+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:56:02.903967+0000","last_undegraded":"2023-10-17T01:56:02.903967+0000","last_fullsized":"2023-10-17T01:56:02.903967+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":9434,"log_dups_size":0,"ondisk_log_size":9434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"102'481920","reported_seq":961245,"reported_epoch":102,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:56:03.499977+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T01:56:03.499977+0000","last_peered":"2023-10-17T01:56:03.499977+0000","last_clean":"2023-10-17T01:56:03.499977+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:56:03.499977+0000","last_undegraded":"2023-10-17T01:56:03.499977+0000","last_fullsized":"2023-10-17T01:56:03.499977+0000","mapping_epoch":16,"log_start":"100'471910","ondisk_log_start":"100'471910","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1170,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 470s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":6947086336,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":480262,"num_read_kb":1889440,"num_write":479960,"num_write_kb":1919840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":255,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-17T01:55:33.276590+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:55:33.276590+0000","last_peered":"2023-10-17T01:55:33.276590+0000","last_clean":"2023-10-17T01:55:33.276590+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:55:33.276590+0000","last_undegraded":"2023-10-17T01:55:33.276590+0000","last_fullsized":"2023-10-17T01:55:33.276590+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"102'463945","reported_seq":925928,"reported_epoch":102,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:56:03.780705+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T01:56:03.780705+0000","last_peered":"2023-10-17T01:56:03.780705+0000","last_clean":"2023-10-17T01:56:03.780705+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:56:03.780705+0000","last_undegraded":"2023-10-17T01:56:03.780705+0000","last_fullsized":"2023-10-17T01:56:03.780705+0000","mapping_epoch":16,"log_start":"100'453918","ondisk_log_start":"100'453918","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1528,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 462s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6680604672,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":462838,"num_read_kb":1822824,"num_write":461830,"num_write_kb":1847320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2651,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-17T01:55:56.796469+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T01:55:56.796469+0000","last_peered":"2023-10-17T01:55:56.796469+0000","last_clean":"2023-10-17T01:55:56.796469+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:55:56.796469+0000","last_undegraded":"2023-10-17T01:55:56.796469+0000","last_fullsized":"2023-10-17T01:55:56.796469+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"102'483253","reported_seq":963917,"reported_epoch":102,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:56:03.494852+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T01:56:03.494852+0000","last_peered":"2023-10-17T01:56:03.494852+0000","last_clean":"2023-10-17T01:56:03.494852+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:56:03.494852+0000","last_undegraded":"2023-10-17T01:56:03.494852+0000","last_fullsized":"2023-10-17T01:56:03.494852+0000","mapping_epoch":16,"log_start":"100'473219","ondisk_log_start":"100'473219","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1436,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 478s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":6964719616,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":481582,"num_read_kb":1895504,"num_write":481150,"num_write_kb":1924600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":3980,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-17T01:55:56.550522+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T01:55:56.550522+0000","last_peered":"2023-10-17T01:55:56.550522+0000","last_clean":"2023-10-17T01:55:56.550522+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:55:56.550522+0000","last_undegraded":"2023-10-17T01:55:56.550522+0000","last_fullsized":"2023-10-17T01:55:56.550522+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"102'471856","reported_seq":943021,"reported_epoch":102,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:56:03.809367+0000","last_change":"2023-10-17T01:56:03.787914+0000","last_active":"2023-10-17T01:56:03.809367+0000","last_peered":"2023-10-17T01:56:03.809367+0000","last_clean":"2023-10-17T01:56:03.809367+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:56:03.809367+0000","last_undegraded":"2023-10-17T01:56:03.809367+0000","last_fullsized":"2023-10-17T01:56:03.809367+0000","mapping_epoch":16,"log_start":"100'461816","ondisk_log_start":"100'461816","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6813650963,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":472127,"num_read_kb":1857748,"num_write":470583,"num_write_kb":1882326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"102'9405","reported_seq":13263,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-17T01:56:02.986042+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T01:56:02.986042+0000","last_peered":"2023-10-17T01:56:02.986042+0000","last_clean":"2023-10-17T01:56:02.986042+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:56:02.986042+0000","last_undegraded":"2023-10-17T01:56:02.986042+0000","last_fullsized":"2023-10-17T01:56:02.986042+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":9405,"log_dups_size":0,"ondisk_log_size":9405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":275,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-17T01:54:42.197794+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:54:42.197794+0000","last_peered":"2023-10-17T01:54:42.197794+0000","last_clean":"2023-10-17T01:54:42.197794+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:54:42.197794+0000","last_undegraded":"2023-10-17T01:54:42.197794+0000","last_fullsized":"2023-10-17T01:54:42.197794+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27406061587,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1896809,"num_read_kb":7465516,"num_write":1893523,"num_write_kb":7574086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15507992576,"data_stored":18908852262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40111,"ondisk_log_size":40111,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":44854,"ondisk_log_size":44854,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739258,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9239488,"kb_used_data":7569848,"kb_used_omap":668,"kb_used_meta":1668963,"kb_avail":85132352,"statfs":{"total":96636764160,"available":87175528448,"internally_reserved":0,"allocated":7751524352,"data_stored":9451365067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":684800,"internal_metadata":1709018368},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,4,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:55:12 2023","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.45400000000000001,"15min":0.39300000000000002},"min":{"1min":0.189,"5min":0.189,"15min":0.16900000000000001},"max":{"1min":2.5790000000000002,"5min":4.2140000000000004,"15min":4.5030000000000001},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.497,"15min":0.502},"min":{"1min":0.19800000000000001,"5min":0.188,"15min":0.188},"max":{"1min":2.5419999999999998,"5min":3.5870000000000002,"15min":5.4240000000000004},"last":0.44500000000000001}]},{"osd":1,"last update":"Tue Oct 17 01:55:35 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.47499999999999998,"15min":0.51500000000000001},"min":{"1min":0.23899999999999999,"5min":0.20699999999999999,"15min":0.19400000000000001},"max":{"1min":0.496,"5min":5.7270000000000003,"15min":12.413},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60599999999999998,"15min":0.55400000000000005},"min":{"1min":0.28999999999999998,"5min":0.24399999999999999,"15min":0.19500000000000001},"max":{"1min":4.8739999999999997,"5min":5.7599999999999998,"15min":5.7599999999999998},"last":0.32800000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739256,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5116724,"kb_used_data":3730796,"kb_used_omap":285,"kb_used_meta":1385634,"kb_avail":89255116,"statfs":{"total":96636764160,"available":91397238784,"internally_reserved":0,"allocated":3820335104,"data_stored":4657945318,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":292849,"internal_metadata":1418889231},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,8,5,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:56:01 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.57099999999999995,"15min":0.54200000000000004},"min":{"1min":0.188,"5min":0.18099999999999999,"15min":0.16200000000000001},"max":{"1min":4.141,"5min":6.6890000000000001,"15min":23.177},"last":0.246},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.621,"15min":0.59299999999999997},"min":{"1min":0.246,"5min":0.246,"15min":0.20200000000000001},"max":{"1min":4.1820000000000004,"5min":6.7539999999999996,"15min":23.207999999999998},"last":0.28499999999999998}]},{"osd":2,"last update":"Tue Oct 17 01:56:01 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.46100000000000002,"15min":0.54200000000000004},"min":{"1min":0.20999999999999999,"5min":0.188,"15min":0.184},"max":{"1min":4.2539999999999996,"5min":4.2539999999999996,"15min":13.757},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.44500000000000001,"15min":0.58399999999999996},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.79400000000000004,"5min":4.1589999999999998,"15min":14.195},"last":0.54800000000000004}]}]},{"osd":0,"up_from":8,"seq":34359739256,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5765740,"kb_used_data":3845088,"kb_used_omap":289,"kb_used_meta":1920350,"kb_avail":88606100,"statfs":{"total":96636764160,"available":90732646400,"internally_reserved":0,"allocated":3937370112,"data_stored":4799825837,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":296403,"internal_metadata":1966438957},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,5,1,6,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:55:35 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.38700000000000001,"15min":0.499},"min":{"1min":0.186,"5min":0.17999999999999999,"15min":0.16200000000000001},"max":{"1min":0.52100000000000002,"5min":4.1680000000000001,"15min":5.9829999999999997},"last":0.20699999999999999},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.48199999999999998,"15min":0.53300000000000003},"min":{"1min":0.192,"5min":0.191,"15min":0.189},"max":{"1min":0.54900000000000004,"5min":5.6909999999999998,"15min":6.0750000000000002},"last":0.22500000000000001}]},{"osd":2,"last update":"Tue Oct 17 01:55:45 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.36199999999999999,"15min":0.35199999999999998},"min":{"1min":0.21299999999999999,"5min":0.191,"15min":0.18099999999999999},"max":{"1min":0.56000000000000005,"5min":2.6120000000000001,"15min":2.8849999999999998},"last":0.378},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.64300000000000002,"15min":0.54800000000000004},"min":{"1min":0.245,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.50800000000000001,"5min":10.378,"15min":10.378},"last":0.26700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3936710656,"data_stored":4799307776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3819175936,"data_stored":4657008659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7752105984,"data_stored":9452535827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:56:06.429 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:56:06.429 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:56:06.430 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:56:06.430 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:56:06.431 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T01:56:26.432 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:56:27.048 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:56:27.049 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:56:27.073 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2329,"stamp":"2023-10-17T01:56:25.432780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27436280664,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1927539,"num_read_kb":7513181,"num_write":1901402,"num_write_kb":7605259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85160,"ondisk_log_size":85160,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19805104,"kb_used_data":15205144,"kb_used_omap":1245,"kb_used_meta":4598690,"kb_avail":263310416,"statfs":{"total":289910292480,"available":269629865984,"internally_reserved":0,"allocated":15570067456,"data_stored":18969970010,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1275732,"internal_metadata":4709058732},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,7,12,11,16,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22265856,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5018,"num_read_kb":20028,"num_write":5450,"num_write_kb":21800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002106"},"pg_stats":[{"pgid":"2.7","version":"102'12525","reported_seq":17761,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-17T01:56:23.347466+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T01:56:23.347466+0000","last_peered":"2023-10-17T01:56:23.347466+0000","last_clean":"2023-10-17T01:56:23.347466+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:56:23.347466+0000","last_undegraded":"2023-10-17T01:56:23.347466+0000","last_fullsized":"2023-10-17T01:56:23.347466+0000","mapping_epoch":13,"log_start":"62'2500","ondisk_log_start":"62'2500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"102'9434","reported_seq":14533,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-17T01:56:23.322049+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T01:56:23.322049+0000","last_peered":"2023-10-17T01:56:23.322049+0000","last_clean":"2023-10-17T01:56:23.322049+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:56:23.322049+0000","last_undegraded":"2023-10-17T01:56:23.322049+0000","last_fullsized":"2023-10-17T01:56:23.322049+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":9434,"log_dups_size":0,"ondisk_log_size":9434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"102'6327","reported_seq":9028,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-17T01:56:23.315577+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T01:56:23.315577+0000","last_peered":"2023-10-17T01:56:23.315577+0000","last_clean":"2023-10-17T01:56:23.315577+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:56:23.315577+0000","last_undegraded":"2023-10-17T01:56:23.315577+0000","last_fullsized":"2023-10-17T01:56:23.315577+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6327,"log_dups_size":0,"ondisk_log_size":6327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"102'9467","reported_seq":13973,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-17T01:56:23.330950+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T01:56:23.330950+0000","last_peered":"2023-10-17T01:56:23.330950+0000","last_clean":"2023-10-17T01:56:23.330950+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:56:23.330950+0000","last_undegraded":"2023-10-17T01:56:23.330950+0000","last_fullsized":"2023-10-17T01:56:23.330950+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":9467,"log_dups_size":0,"ondisk_log_size":9467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"102'483748","reported_seq":964773,"reported_epoch":102,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:56:23.668569+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T01:56:23.668569+0000","last_peered":"2023-10-17T01:56:23.668569+0000","last_clean":"2023-10-17T01:56:23.668569+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:56:23.668569+0000","last_undegraded":"2023-10-17T01:56:23.668569+0000","last_fullsized":"2023-10-17T01:56:23.668569+0000","mapping_epoch":16,"log_start":"100'473710","ondisk_log_start":"100'473710","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1204,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 490s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":6954455040,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":481954,"num_read_kb":1896192,"num_write":481768,"num_write_kb":1927072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":255,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-17T01:55:33.276590+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:55:33.276590+0000","last_peered":"2023-10-17T01:55:33.276590+0000","last_clean":"2023-10-17T01:55:33.276590+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:55:33.276590+0000","last_undegraded":"2023-10-17T01:55:33.276590+0000","last_fullsized":"2023-10-17T01:55:33.276590+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"102'465743","reported_seq":929376,"reported_epoch":102,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:56:23.948009+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T01:56:23.948009+0000","last_peered":"2023-10-17T01:56:23.948009+0000","last_clean":"2023-10-17T01:56:23.948009+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:56:23.948009+0000","last_undegraded":"2023-10-17T01:56:23.948009+0000","last_fullsized":"2023-10-17T01:56:23.948009+0000","mapping_epoch":16,"log_start":"100'455718","ondisk_log_start":"100'455718","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1582,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 482s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6687866880,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":464479,"num_read_kb":1829372,"num_write":463601,"num_write_kb":1854404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2660,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-17T01:56:17.066408+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T01:56:17.066408+0000","last_peered":"2023-10-17T01:56:17.066408+0000","last_clean":"2023-10-17T01:56:17.066408+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:56:17.066408+0000","last_undegraded":"2023-10-17T01:56:17.066408+0000","last_fullsized":"2023-10-17T01:56:17.066408+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"102'485143","reported_seq":967548,"reported_epoch":102,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:56:23.672878+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T01:56:23.672878+0000","last_peered":"2023-10-17T01:56:23.672878+0000","last_clean":"2023-10-17T01:56:23.672878+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:56:23.672878+0000","last_undegraded":"2023-10-17T01:56:23.672878+0000","last_fullsized":"2023-10-17T01:56:23.672878+0000","mapping_epoch":16,"log_start":"100'475120","ondisk_log_start":"100'475120","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1504,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 498s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":6972289024,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":483315,"num_read_kb":1902416,"num_write":483004,"num_write_kb":1932016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":3993,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-17T01:56:17.023485+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T01:56:17.023485+0000","last_peered":"2023-10-17T01:56:17.023485+0000","last_clean":"2023-10-17T01:56:17.023485+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:56:17.023485+0000","last_undegraded":"2023-10-17T01:56:17.023485+0000","last_fullsized":"2023-10-17T01:56:17.023485+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"102'473672","reported_seq":946545,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-17T01:56:23.954448+0000","last_change":"2023-10-17T01:56:23.895290+0000","last_active":"2023-10-17T01:56:23.954448+0000","last_peered":"2023-10-17T01:56:23.954448+0000","last_clean":"2023-10-17T01:56:23.954448+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:56:23.954448+0000","last_undegraded":"2023-10-17T01:56:23.954448+0000","last_fullsized":"2023-10-17T01:56:23.954448+0000","mapping_epoch":16,"log_start":"100'463616","ondisk_log_start":"100'463616","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6821077011,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":473829,"num_read_kb":1864548,"num_write":472399,"num_write_kb":1889590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"102'9438","reported_seq":13308,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-17T01:56:23.367297+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T01:56:23.367297+0000","last_peered":"2023-10-17T01:56:23.367297+0000","last_clean":"2023-10-17T01:56:23.367297+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:56:23.367297+0000","last_undegraded":"2023-10-17T01:56:23.367297+0000","last_fullsized":"2023-10-17T01:56:23.367297+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":9438,"log_dups_size":0,"ondisk_log_size":9438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":275,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-17T01:54:42.197794+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:54:42.197794+0000","last_peered":"2023-10-17T01:54:42.197794+0000","last_clean":"2023-10-17T01:54:42.197794+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:54:42.197794+0000","last_undegraded":"2023-10-17T01:54:42.197794+0000","last_fullsized":"2023-10-17T01:54:42.197794+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27435687955,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1903577,"num_read_kb":7492528,"num_write":1900772,"num_write_kb":7603082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15567400960,"data_stored":18968260646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40142,"ondisk_log_size":40142,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":44919,"ondisk_log_size":44919,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739262,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9366976,"kb_used_data":7600248,"kb_used_omap":668,"kb_used_meta":1766051,"kb_avail":85004864,"statfs":{"total":96636764160,"available":87044980736,"internally_reserved":0,"allocated":7782653952,"data_stored":9482490567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":684800,"internal_metadata":1808436480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:56:14 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.47099999999999997,"15min":0.39800000000000002},"min":{"1min":0.20999999999999999,"5min":0.189,"15min":0.16900000000000001},"max":{"1min":0.86299999999999999,"5min":4.2140000000000004,"15min":4.5030000000000001},"last":0.307},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.55400000000000005,"15min":0.499},"min":{"1min":0.249,"5min":0.188,"15min":0.188},"max":{"1min":5.3280000000000003,"5min":5.3280000000000003,"15min":5.3280000000000003},"last":0.79000000000000004}]},{"osd":1,"last update":"Tue Oct 17 01:55:35 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.47499999999999998,"15min":0.51500000000000001},"min":{"1min":0.23899999999999999,"5min":0.20699999999999999,"15min":0.19400000000000001},"max":{"1min":0.496,"5min":5.7270000000000003,"15min":12.413},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60599999999999998,"15min":0.55400000000000005},"min":{"1min":0.28999999999999998,"5min":0.24399999999999999,"15min":0.19500000000000001},"max":{"1min":4.8739999999999997,"5min":5.7599999999999998,"15min":5.7599999999999998},"last":0.28999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739260,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5162944,"kb_used_data":3745080,"kb_used_omap":285,"kb_used_meta":1417570,"kb_avail":89208896,"statfs":{"total":96636764160,"available":91349909504,"internally_reserved":0,"allocated":3834961920,"data_stored":4672572134,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":292849,"internal_metadata":1451591695},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:56:01 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.57099999999999995,"15min":0.54200000000000004},"min":{"1min":0.188,"5min":0.18099999999999999,"15min":0.16200000000000001},"max":{"1min":4.141,"5min":6.6890000000000001,"15min":23.177},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.621,"15min":0.59299999999999997},"min":{"1min":0.246,"5min":0.246,"15min":0.20200000000000001},"max":{"1min":4.1820000000000004,"5min":6.7539999999999996,"15min":23.207999999999998},"last":0.85199999999999998}]},{"osd":2,"last update":"Tue Oct 17 01:56:01 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.46100000000000002,"15min":0.54200000000000004},"min":{"1min":0.20999999999999999,"5min":0.188,"15min":0.184},"max":{"1min":4.2539999999999996,"5min":4.2539999999999996,"15min":13.757},"last":0.437},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.44500000000000001,"15min":0.58399999999999996},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.79400000000000004,"5min":4.1589999999999998,"15min":14.195},"last":0.53700000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739260,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5275184,"kb_used_data":3859816,"kb_used_omap":291,"kb_used_meta":1415068,"kb_avail":89096656,"statfs":{"total":96636764160,"available":91234975744,"internally_reserved":0,"allocated":3952451584,"data_stored":4814907309,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":298083,"internal_metadata":1449030557},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,2,6,4,8,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:55:35 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.38700000000000001,"15min":0.499},"min":{"1min":0.186,"5min":0.17999999999999999,"15min":0.16200000000000001},"max":{"1min":0.52100000000000002,"5min":4.1680000000000001,"15min":5.9829999999999997},"last":0.218},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.48199999999999998,"15min":0.53300000000000003},"min":{"1min":0.192,"5min":0.191,"15min":0.189},"max":{"1min":0.54900000000000004,"5min":5.6909999999999998,"15min":6.0750000000000002},"last":0.30499999999999999}]},{"osd":2,"last update":"Tue Oct 17 01:55:45 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.36199999999999999,"15min":0.35199999999999998},"min":{"1min":0.21299999999999999,"5min":0.191,"15min":0.18099999999999999},"max":{"1min":0.56000000000000005,"5min":2.6120000000000001,"15min":2.8849999999999998},"last":0.25},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.64300000000000002,"15min":0.54800000000000004},"min":{"1min":0.245,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.50800000000000001,"5min":10.378,"15min":10.378},"last":0.27200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3951714304,"data_stored":4814311424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3833864192,"data_stored":4671696915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7781822464,"data_stored":9482252307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:56:27.075 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:56:27.075 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:56:27.075 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:56:27.076 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:56:27.076 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T01:56:30.690 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:56:30.686+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:56:30.703 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:56:30.702+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:56:47.077 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:56:47.592 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:56:47.593 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:56:47.645 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2340,"stamp":"2023-10-17T01:56:47.436479+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27466287960,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1934423,"num_read_kb":7540677,"num_write":1908729,"num_write_kb":7634567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85414,"ondisk_log_size":85414,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20105992,"kb_used_data":15269996,"kb_used_omap":1245,"kb_used_meta":4834722,"kb_avail":263009528,"statfs":{"total":289910292480,"available":269321756672,"internally_reserved":0,"allocated":15636475904,"data_stored":19036378458,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1275732,"internal_metadata":4950755500},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,6,5,12,19,15,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15040512,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3357,"num_read_kb":13408,"num_write":3671,"num_write_kb":14684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001964"},"pg_stats":[{"pgid":"2.7","version":"102'12565","reported_seq":17817,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-17T01:56:42.623661+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T01:56:42.623661+0000","last_peered":"2023-10-17T01:56:42.623661+0000","last_clean":"2023-10-17T01:56:42.623661+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:56:42.623661+0000","last_undegraded":"2023-10-17T01:56:42.623661+0000","last_fullsized":"2023-10-17T01:56:42.623661+0000","mapping_epoch":13,"log_start":"62'2500","ondisk_log_start":"62'2500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"102'9465","reported_seq":14576,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-17T01:56:42.623573+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T01:56:42.623573+0000","last_peered":"2023-10-17T01:56:42.623573+0000","last_clean":"2023-10-17T01:56:42.623573+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:56:42.623573+0000","last_undegraded":"2023-10-17T01:56:42.623573+0000","last_fullsized":"2023-10-17T01:56:42.623573+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":9465,"log_dups_size":0,"ondisk_log_size":9465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"102'6347","reported_seq":9056,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-17T01:56:42.623617+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T01:56:42.623617+0000","last_peered":"2023-10-17T01:56:42.623617+0000","last_clean":"2023-10-17T01:56:42.623617+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:56:42.623617+0000","last_undegraded":"2023-10-17T01:56:42.623617+0000","last_fullsized":"2023-10-17T01:56:42.623617+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6347,"log_dups_size":0,"ondisk_log_size":6347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"102'9498","reported_seq":14016,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-17T01:56:42.623439+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T01:56:42.623439+0000","last_peered":"2023-10-17T01:56:42.623439+0000","last_clean":"2023-10-17T01:56:42.623439+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:56:42.623439+0000","last_undegraded":"2023-10-17T01:56:42.623439+0000","last_fullsized":"2023-10-17T01:56:42.623439+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":9498,"log_dups_size":0,"ondisk_log_size":9498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"102'485650","reported_seq":968437,"reported_epoch":102,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:56:43.810508+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T01:56:43.810508+0000","last_peered":"2023-10-17T01:56:43.810508+0000","last_clean":"2023-10-17T01:56:43.810508+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:56:43.810508+0000","last_undegraded":"2023-10-17T01:56:43.810508+0000","last_fullsized":"2023-10-17T01:56:43.810508+0000","mapping_epoch":16,"log_start":"101'475610","ondisk_log_start":"101'475610","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1240,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 510s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":6962171904,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":483706,"num_read_kb":1903196,"num_write":483649,"num_write_kb":1934596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":255,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-17T01:55:33.276590+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:55:33.276590+0000","last_peered":"2023-10-17T01:55:33.276590+0000","last_clean":"2023-10-17T01:55:33.276590+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:55:33.276590+0000","last_undegraded":"2023-10-17T01:55:33.276590+0000","last_fullsized":"2023-10-17T01:55:33.276590+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"102'467586","reported_seq":932929,"reported_epoch":102,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:56:44.028135+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T01:56:44.028135+0000","last_peered":"2023-10-17T01:56:44.028135+0000","last_clean":"2023-10-17T01:56:44.028135+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:56:44.028135+0000","last_undegraded":"2023-10-17T01:56:44.028135+0000","last_fullsized":"2023-10-17T01:56:44.028135+0000","mapping_epoch":16,"log_start":"101'457518","ondisk_log_start":"101'457518","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1634,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 502s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6695301120,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":466179,"num_read_kb":1836164,"num_write":465416,"num_write_kb":1861664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2670,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-17T01:56:42.636893+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T01:56:42.636893+0000","last_peered":"2023-10-17T01:56:42.636893+0000","last_clean":"2023-10-17T01:56:42.636893+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:56:42.636893+0000","last_undegraded":"2023-10-17T01:56:42.636893+0000","last_fullsized":"2023-10-17T01:56:42.636893+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"102'487003","reported_seq":971091,"reported_epoch":102,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:56:43.787692+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T01:56:43.787692+0000","last_peered":"2023-10-17T01:56:43.787692+0000","last_clean":"2023-10-17T01:56:43.787692+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:56:43.787692+0000","last_undegraded":"2023-10-17T01:56:43.787692+0000","last_fullsized":"2023-10-17T01:56:43.787692+0000","mapping_epoch":16,"log_start":"101'476921","ondisk_log_start":"101'476921","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1554,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 518s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":6979780608,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":484991,"num_read_kb":1909112,"num_write":484838,"num_write_kb":1939352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4009,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-17T01:56:42.662897+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T01:56:42.662897+0000","last_peered":"2023-10-17T01:56:42.662897+0000","last_clean":"2023-10-17T01:56:42.662897+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:56:42.662897+0000","last_undegraded":"2023-10-17T01:56:42.662897+0000","last_fullsized":"2023-10-17T01:56:42.662897+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"102'475469","reported_seq":950110,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-17T01:56:44.030941+0000","last_change":"2023-10-17T01:56:43.297489+0000","last_active":"2023-10-17T01:56:44.030941+0000","last_peered":"2023-10-17T01:56:44.030941+0000","last_clean":"2023-10-17T01:56:44.030941+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:56:44.030941+0000","last_undegraded":"2023-10-17T01:56:44.030941+0000","last_fullsized":"2023-10-17T01:56:44.030941+0000","mapping_epoch":16,"log_start":"101'465416","ondisk_log_start":"101'465416","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6828441619,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":475585,"num_read_kb":1871552,"num_write":474196,"num_write_kb":1896778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"102'9469","reported_seq":13351,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-17T01:56:42.623427+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T01:56:42.623427+0000","last_peered":"2023-10-17T01:56:42.623427+0000","last_clean":"2023-10-17T01:56:42.623427+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:56:42.623427+0000","last_undegraded":"2023-10-17T01:56:42.623427+0000","last_fullsized":"2023-10-17T01:56:42.623427+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":9469,"log_dups_size":0,"ondisk_log_size":9469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":276,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-17T01:56:02.191744+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:56:02.191744+0000","last_peered":"2023-10-17T01:56:02.191744+0000","last_clean":"2023-10-17T01:56:02.191744+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:56:02.191744+0000","last_undegraded":"2023-10-17T01:56:02.191744+0000","last_fullsized":"2023-10-17T01:56:02.191744+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27465695251,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1910461,"num_read_kb":7520024,"num_write":1908099,"num_write_kb":7632390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15634788352,"data_stored":19035648038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40243,"ondisk_log_size":40243,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":45072,"ondisk_log_size":45072,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739267,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9529272,"kb_used_data":7636208,"kb_used_omap":668,"kb_used_meta":1892387,"kb_avail":84842568,"statfs":{"total":96636764160,"available":86878789632,"internally_reserved":0,"allocated":7819476992,"data_stored":9519313607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":684800,"internal_metadata":1937804544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,5,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:56:14 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.47099999999999997,"15min":0.39800000000000002},"min":{"1min":0.20999999999999999,"5min":0.189,"15min":0.16900000000000001},"max":{"1min":0.86299999999999999,"5min":4.2140000000000004,"15min":4.5030000000000001},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.55400000000000005,"15min":0.499},"min":{"1min":0.249,"5min":0.188,"15min":0.188},"max":{"1min":5.3280000000000003,"5min":5.3280000000000003,"15min":5.3280000000000003},"last":0.36799999999999999}]},{"osd":1,"last update":"Tue Oct 17 01:56:40 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.51000000000000001,"15min":0.54900000000000004},"min":{"1min":0.22600000000000001,"5min":0.20699999999999999,"15min":0.19400000000000001},"max":{"1min":9.5429999999999993,"5min":9.5429999999999993,"15min":12.413},"last":0.252},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.59899999999999998,"15min":0.58699999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.19500000000000001},"max":{"1min":9.5999999999999996,"5min":9.5999999999999996,"15min":9.5999999999999996},"last":0.38900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739264,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5229656,"kb_used_data":3759052,"kb_used_omap":285,"kb_used_meta":1470306,"kb_avail":89142184,"statfs":{"total":96636764160,"available":91281596416,"internally_reserved":0,"allocated":3849269248,"data_stored":4686879462,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":292849,"internal_metadata":1505593359},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,3,2,3,7,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:56:01 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.57099999999999995,"15min":0.54200000000000004},"min":{"1min":0.188,"5min":0.18099999999999999,"15min":0.16200000000000001},"max":{"1min":4.141,"5min":6.6890000000000001,"15min":23.177},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.621,"15min":0.59299999999999997},"min":{"1min":0.246,"5min":0.246,"15min":0.20200000000000001},"max":{"1min":4.1820000000000004,"5min":6.7539999999999996,"15min":23.207999999999998},"last":0.40100000000000002}]},{"osd":2,"last update":"Tue Oct 17 01:56:01 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.46100000000000002,"15min":0.54200000000000004},"min":{"1min":0.20999999999999999,"5min":0.188,"15min":0.184},"max":{"1min":4.2539999999999996,"5min":4.2539999999999996,"15min":13.757},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.44500000000000001,"15min":0.58399999999999996},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.79400000000000004,"5min":4.1589999999999998,"15min":14.195},"last":0.45300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739264,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5347064,"kb_used_data":3874736,"kb_used_omap":291,"kb_used_meta":1472028,"kb_avail":89024776,"statfs":{"total":96636764160,"available":91161370624,"internally_reserved":0,"allocated":3967729664,"data_stored":4830185389,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":298083,"internal_metadata":1507357597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,4,4,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.39100000000000001,"15min":0.48299999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.16200000000000001},"max":{"1min":0.90700000000000003,"5min":4.1680000000000001,"15min":5.9829999999999997},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.48499999999999999,"15min":0.52000000000000002},"min":{"1min":0.22500000000000001,"5min":0.191,"15min":0.189},"max":{"1min":0.71499999999999997,"5min":5.6909999999999998,"15min":6.0750000000000002},"last":1.5429999999999999}]},{"osd":2,"last update":"Tue Oct 17 01:55:45 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.36199999999999999,"15min":0.35199999999999998},"min":{"1min":0.21299999999999999,"5min":0.191,"15min":0.18099999999999999},"max":{"1min":0.56000000000000005,"5min":2.6120000000000001,"15min":2.8849999999999998},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.64300000000000002,"15min":0.54800000000000004},"min":{"1min":0.245,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.50800000000000001,"5min":10.378,"15min":10.378},"last":1.6839999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3966926848,"data_stored":4829523968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3848654848,"data_stored":4686487571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7819206656,"data_stored":9519636499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:56:47.647 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:56:47.648 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:56:47.648 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:56:47.649 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:56:47.649 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-17T01:56:48.212 INFO:teuthology.orchestra.run.smithi169.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-17T01:56:48.243 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-17T01:56:48.834 INFO:teuthology.orchestra.run.smithi169.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-17T01:56:48.873 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-17T01:56:49.401 INFO:teuthology.orchestra.run.smithi169.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-17T01:56:49.438 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:56:49.433+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:56:49.461 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-17T01:56:49.467 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:56:49.465+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:56:49.468 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:56:49.465+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:56:49.468 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:56:49.465+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:56:49.968 INFO:teuthology.orchestra.run.smithi169.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-17T01:56:50.010 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T01:57:10.011 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:57:10.629 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:57:10.629 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:57:10.679 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2351,"stamp":"2023-10-17T01:57:09.441400+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27502324568,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1943010,"num_read_kb":7574965,"num_write":1917578,"num_write_kb":7669963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85564,"ondisk_log_size":85564,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20370364,"kb_used_data":15333020,"kb_used_omap":1245,"kb_used_meta":5036066,"kb_avail":262745156,"statfs":{"total":289910292480,"available":269051039744,"internally_reserved":0,"allocated":15701012480,"data_stored":19100915034,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1275732,"internal_metadata":5156931756},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,5,16,15,16,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21962752,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5322,"num_read_kb":21236,"num_write":5389,"num_write_kb":21556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001933"},"pg_stats":[{"pgid":"2.7","version":"102'12612","reported_seq":17884,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-17T01:57:08.224354+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T01:57:08.224354+0000","last_peered":"2023-10-17T01:57:08.224354+0000","last_clean":"2023-10-17T01:57:08.224354+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:57:08.224354+0000","last_undegraded":"2023-10-17T01:57:08.224354+0000","last_fullsized":"2023-10-17T01:57:08.224354+0000","mapping_epoch":13,"log_start":"62'2600","ondisk_log_start":"62'2600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"102'9501","reported_seq":14627,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-17T01:57:08.223766+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T01:57:08.223766+0000","last_peered":"2023-10-17T01:57:08.223766+0000","last_clean":"2023-10-17T01:57:08.223766+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:57:08.223766+0000","last_undegraded":"2023-10-17T01:57:08.223766+0000","last_fullsized":"2023-10-17T01:57:08.223766+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":9501,"log_dups_size":0,"ondisk_log_size":9501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"102'6371","reported_seq":9090,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-17T01:57:08.223193+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T01:57:08.223193+0000","last_peered":"2023-10-17T01:57:08.223193+0000","last_clean":"2023-10-17T01:57:08.223193+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:57:08.223193+0000","last_undegraded":"2023-10-17T01:57:08.223193+0000","last_fullsized":"2023-10-17T01:57:08.223193+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6371,"log_dups_size":0,"ondisk_log_size":6371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"102'9534","reported_seq":14067,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-17T01:57:08.223288+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T01:57:08.223288+0000","last_peered":"2023-10-17T01:57:08.223288+0000","last_clean":"2023-10-17T01:57:08.223288+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:57:08.223288+0000","last_undegraded":"2023-10-17T01:57:08.223288+0000","last_fullsized":"2023-10-17T01:57:08.223288+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":9534,"log_dups_size":0,"ondisk_log_size":9534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"102'487868","reported_seq":972889,"reported_epoch":102,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:57:08.957251+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T01:57:08.957251+0000","last_peered":"2023-10-17T01:57:08.957251+0000","last_clean":"2023-10-17T01:57:08.957251+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:57:08.957251+0000","last_undegraded":"2023-10-17T01:57:08.957251+0000","last_fullsized":"2023-10-17T01:57:08.957251+0000","mapping_epoch":16,"log_start":"101'477810","ondisk_log_start":"101'477810","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1304,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 535s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":6971117568,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":485927,"num_read_kb":1912072,"num_write":485834,"num_write_kb":1943336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":255,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-17T01:55:33.276590+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:55:33.276590+0000","last_peered":"2023-10-17T01:55:33.276590+0000","last_clean":"2023-10-17T01:55:33.276590+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:55:33.276590+0000","last_undegraded":"2023-10-17T01:55:33.276590+0000","last_fullsized":"2023-10-17T01:55:33.276590+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"102'469810","reported_seq":937271,"reported_epoch":102,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:57:09.209578+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T01:57:09.209578+0000","last_peered":"2023-10-17T01:57:09.209578+0000","last_clean":"2023-10-17T01:57:09.209578+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:57:09.209578+0000","last_undegraded":"2023-10-17T01:57:09.209578+0000","last_fullsized":"2023-10-17T01:57:09.209578+0000","mapping_epoch":16,"log_start":"101'459718","ondisk_log_start":"101'459718","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1716,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 527s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6704160768,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":468282,"num_read_kb":1844556,"num_write":467595,"num_write_kb":1870380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2678,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-17T01:57:03.104214+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T01:57:03.104214+0000","last_peered":"2023-10-17T01:57:03.104214+0000","last_clean":"2023-10-17T01:57:03.104214+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:57:03.104214+0000","last_undegraded":"2023-10-17T01:57:03.104214+0000","last_fullsized":"2023-10-17T01:57:03.104214+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"102'489310","reported_seq":975514,"reported_epoch":102,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:57:08.973400+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T01:57:08.973400+0000","last_peered":"2023-10-17T01:57:08.973400+0000","last_clean":"2023-10-17T01:57:08.973400+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:57:08.973400+0000","last_undegraded":"2023-10-17T01:57:08.973400+0000","last_fullsized":"2023-10-17T01:57:08.973400+0000","mapping_epoch":16,"log_start":"101'479222","ondisk_log_start":"101'479222","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1636,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 543s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":6988988416,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":487092,"num_read_kb":1917500,"num_write":487100,"num_write_kb":1948400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4021,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-17T01:57:03.103928+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T01:57:03.103928+0000","last_peered":"2023-10-17T01:57:03.103928+0000","last_clean":"2023-10-17T01:57:03.103928+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:57:03.103928+0000","last_undegraded":"2023-10-17T01:57:03.103928+0000","last_fullsized":"2023-10-17T01:57:03.103928+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"102'477692","reported_seq":954501,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-17T01:57:09.213400+0000","last_change":"2023-10-17T01:57:07.560406+0000","last_active":"2023-10-17T01:57:09.213400+0000","last_peered":"2023-10-17T01:57:09.213400+0000","last_clean":"2023-10-17T01:57:09.213400+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:57:09.213400+0000","last_undegraded":"2023-10-17T01:57:09.213400+0000","last_fullsized":"2023-10-17T01:57:09.213400+0000","mapping_epoch":16,"log_start":"101'467616","ondisk_log_start":"101'467616","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6837465107,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":477747,"num_read_kb":1880184,"num_write":476419,"num_write_kb":1905670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"102'9505","reported_seq":13402,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-17T01:57:08.224713+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T01:57:08.224713+0000","last_peered":"2023-10-17T01:57:08.224713+0000","last_clean":"2023-10-17T01:57:08.224713+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:57:08.224713+0000","last_undegraded":"2023-10-17T01:57:08.224713+0000","last_fullsized":"2023-10-17T01:57:08.224713+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":9505,"log_dups_size":0,"ondisk_log_size":9505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":276,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-17T01:56:02.191744+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:56:02.191744+0000","last_peered":"2023-10-17T01:56:02.191744+0000","last_clean":"2023-10-17T01:56:02.191744+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:56:02.191744+0000","last_undegraded":"2023-10-17T01:56:02.191744+0000","last_fullsized":"2023-10-17T01:56:02.191744+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27501731859,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1919048,"num_read_kb":7554312,"num_write":1916948,"num_write_kb":7667786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15700000768,"data_stored":19100860454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40314,"ondisk_log_size":40314,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":45151,"ondisk_log_size":45151,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739272,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9655576,"kb_used_data":7663948,"kb_used_omap":668,"kb_used_meta":1990947,"kb_avail":84716264,"statfs":{"total":96636764160,"available":86749454336,"internally_reserved":0,"allocated":7847882752,"data_stored":9547719367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":684800,"internal_metadata":2038729984},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,4,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:56:14 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.47099999999999997,"15min":0.39800000000000002},"min":{"1min":0.20999999999999999,"5min":0.189,"15min":0.16900000000000001},"max":{"1min":0.86299999999999999,"5min":4.2140000000000004,"15min":4.5030000000000001},"last":0.223},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.55400000000000005,"15min":0.499},"min":{"1min":0.249,"5min":0.188,"15min":0.188},"max":{"1min":5.3280000000000003,"5min":5.3280000000000003,"15min":5.3280000000000003},"last":0.38200000000000001}]},{"osd":1,"last update":"Tue Oct 17 01:56:40 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.51000000000000001,"15min":0.54900000000000004},"min":{"1min":0.22600000000000001,"5min":0.20699999999999999,"15min":0.19400000000000001},"max":{"1min":9.5429999999999993,"5min":9.5429999999999993,"15min":12.413},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.59899999999999998,"15min":0.58699999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.19500000000000001},"max":{"1min":9.5999999999999996,"5min":9.5999999999999996,"15min":9.5999999999999996},"last":0.32900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739269,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5284848,"kb_used_data":3776740,"kb_used_omap":285,"kb_used_meta":1507810,"kb_avail":89086992,"statfs":{"total":96636764160,"available":91225079808,"internally_reserved":0,"allocated":3867381760,"data_stored":4704991974,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":292849,"internal_metadata":1543997455},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,7,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.51600000000000001,"15min":0.53900000000000003},"min":{"1min":0.22900000000000001,"5min":0.188,"15min":0.16200000000000001},"max":{"1min":1.3460000000000001,"5min":6.6890000000000001,"15min":23.177},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.56499999999999995,"15min":0.58999999999999997},"min":{"1min":0.28100000000000003,"5min":0.246,"15min":0.20200000000000001},"max":{"1min":1.3899999999999999,"5min":6.7539999999999996,"15min":23.207999999999998},"last":0.315}]},{"osd":2,"last update":"Tue Oct 17 01:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.48699999999999999,"15min":0.55600000000000005},"min":{"1min":0.26400000000000001,"5min":0.188,"15min":0.184},"max":{"1min":4.194,"5min":4.2539999999999996,"15min":13.757},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.745,"5min":0.46200000000000002,"15min":0.59499999999999997},"min":{"1min":0.23999999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.0010000000000003,"5min":4.0010000000000003,"15min":14.195},"last":0.29799999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739269,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5429940,"kb_used_data":3892332,"kb_used_omap":291,"kb_used_meta":1537308,"kb_avail":88941900,"statfs":{"total":96636764160,"available":91076505600,"internally_reserved":0,"allocated":3985747968,"data_stored":4848203693,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":298083,"internal_metadata":1574204317},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,2,8,4,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.39100000000000001,"15min":0.48299999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.16200000000000001},"max":{"1min":0.90700000000000003,"5min":4.1680000000000001,"15min":5.9829999999999997},"last":0.17999999999999999},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.48499999999999999,"15min":0.52000000000000002},"min":{"1min":0.22500000000000001,"5min":0.191,"15min":0.189},"max":{"1min":0.71499999999999997,"5min":5.6909999999999998,"15min":6.0750000000000002},"last":0.249}]},{"osd":2,"last update":"Tue Oct 17 01:56:46 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.38,"15min":0.35699999999999998},"min":{"1min":0.20300000000000001,"5min":0.191,"15min":0.18099999999999999},"max":{"1min":1.7230000000000001,"5min":2.6120000000000001,"15min":2.8849999999999998},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.68799999999999994,"15min":0.56100000000000005},"min":{"1min":0.23699999999999999,"5min":0.222,"15min":0.19400000000000001},"max":{"1min":1.738,"5min":10.378,"15min":10.378},"last":0.41699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3985137664,"data_stored":4847734784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3866681344,"data_stored":4704514067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7848181760,"data_stored":9548611603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:57:10.681 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:57:10.681 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:57:10.682 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:57:10.682 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:57:10.683 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T01:57:26.510 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:57:26.505+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:57:26.519 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:57:26.517+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbda750 2023-10-17T01:57:26.520 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:57:26.517+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:57:26.530 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:57:26.525+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:57:26.530 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:57:26.525+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:57:26.531 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:57:26.525+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:57:26.531 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:57:26.525+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:57:26.531 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:57:26.525+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:57:26.531 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:57:26.525+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:57:26.532 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:57:26.525+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:57:26.532 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:57:26.525+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:57:26.532 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:57:26.525+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:57:26.533 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:57:26.525+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:57:26.533 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:57:26.525+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:57:30.683 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:57:31.276 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:57:31.277 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:57:31.315 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2362,"stamp":"2023-10-17T01:57:29.445546+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27531504472,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1950186,"num_read_kb":7603601,"num_write":1924732,"num_write_kb":7698579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85591,"ondisk_log_size":85591,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20151544,"kb_used_data":15390432,"kb_used_omap":1247,"kb_used_meta":4759840,"kb_avail":262963976,"statfs":{"total":289910292480,"available":269275111424,"internally_reserved":0,"allocated":15759802368,"data_stored":19159692493,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1277412,"internal_metadata":4874076700},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,5,4,6,8,14,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21925888,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5415,"num_read_kb":21604,"num_write":5376,"num_write_kb":21504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002157"},"pg_stats":[{"pgid":"2.7","version":"103'12652","reported_seq":17941,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-17T01:57:28.703524+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T01:57:28.703524+0000","last_peered":"2023-10-17T01:57:28.703524+0000","last_clean":"2023-10-17T01:57:28.703524+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:57:28.703524+0000","last_undegraded":"2023-10-17T01:57:28.703524+0000","last_fullsized":"2023-10-17T01:57:28.703524+0000","mapping_epoch":13,"log_start":"62'2600","ondisk_log_start":"62'2600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"103'9531","reported_seq":14670,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-17T01:57:28.707086+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T01:57:28.707086+0000","last_peered":"2023-10-17T01:57:28.707086+0000","last_clean":"2023-10-17T01:57:28.707086+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:57:28.707086+0000","last_undegraded":"2023-10-17T01:57:28.707086+0000","last_fullsized":"2023-10-17T01:57:28.707086+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":9531,"log_dups_size":0,"ondisk_log_size":9531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"103'6391","reported_seq":9119,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-17T01:57:28.703144+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T01:57:28.703144+0000","last_peered":"2023-10-17T01:57:28.703144+0000","last_clean":"2023-10-17T01:57:28.703144+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:57:28.703144+0000","last_undegraded":"2023-10-17T01:57:28.703144+0000","last_fullsized":"2023-10-17T01:57:28.703144+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6391,"log_dups_size":0,"ondisk_log_size":6391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"103'9564","reported_seq":14110,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-17T01:57:28.703309+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T01:57:28.703309+0000","last_peered":"2023-10-17T01:57:28.703309+0000","last_clean":"2023-10-17T01:57:28.703309+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:57:28.703309+0000","last_undegraded":"2023-10-17T01:57:28.703309+0000","last_fullsized":"2023-10-17T01:57:28.703309+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":9564,"log_dups_size":0,"ondisk_log_size":9564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"103'489786","reported_seq":976621,"reported_epoch":103,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:57:29.136405+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T01:57:29.136405+0000","last_peered":"2023-10-17T01:57:29.136405+0000","last_clean":"2023-10-17T01:57:29.136405+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:57:29.136405+0000","last_undegraded":"2023-10-17T01:57:29.136405+0000","last_fullsized":"2023-10-17T01:57:29.136405+0000","mapping_epoch":16,"log_start":"101'479710","ondisk_log_start":"101'479710","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1384,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 555s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":6978785280,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":487729,"num_read_kb":1919272,"num_write":487704,"num_write_kb":1950816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":256,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-17T01:56:02.188991+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:56:02.188991+0000","last_peered":"2023-10-17T01:56:02.188991+0000","last_clean":"2023-10-17T01:56:02.188991+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:56:02.188991+0000","last_undegraded":"2023-10-17T01:56:02.188991+0000","last_fullsized":"2023-10-17T01:56:02.188991+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"103'471527","reported_seq":940749,"reported_epoch":103,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:57:29.324740+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T01:57:29.324740+0000","last_peered":"2023-10-17T01:57:29.324740+0000","last_clean":"2023-10-17T01:57:29.324740+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:57:29.324740+0000","last_undegraded":"2023-10-17T01:57:29.324740+0000","last_fullsized":"2023-10-17T01:57:29.324740+0000","mapping_epoch":16,"log_start":"101'461519","ondisk_log_start":"101'461519","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1778,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 547s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6710992896,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":470033,"num_read_kb":1851552,"num_write":469275,"num_write_kb":1877100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2687,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-17T01:57:23.592112+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T01:57:23.592112+0000","last_peered":"2023-10-17T01:57:23.592112+0000","last_clean":"2023-10-17T01:57:23.592112+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:57:23.592112+0000","last_undegraded":"2023-10-17T01:57:23.592112+0000","last_fullsized":"2023-10-17T01:57:23.592112+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"103'491197","reported_seq":979265,"reported_epoch":103,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:57:29.136664+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T01:57:29.136664+0000","last_peered":"2023-10-17T01:57:29.136664+0000","last_clean":"2023-10-17T01:57:29.136664+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:57:29.136664+0000","last_undegraded":"2023-10-17T01:57:29.136664+0000","last_fullsized":"2023-10-17T01:57:29.136664+0000","mapping_epoch":16,"log_start":"101'481123","ondisk_log_start":"101'481123","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1696,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 564s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":6996463616,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":488944,"num_read_kb":1924888,"num_write":488947,"num_write_kb":1955788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4034,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-17T01:57:23.587985+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T01:57:23.587985+0000","last_peered":"2023-10-17T01:57:23.587985+0000","last_clean":"2023-10-17T01:57:23.587985+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:57:23.587985+0000","last_undegraded":"2023-10-17T01:57:23.587985+0000","last_fullsized":"2023-10-17T01:57:23.587985+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"103'479449","reported_seq":958035,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-17T01:57:29.317949+0000","last_change":"2023-10-17T01:57:28.832499+0000","last_active":"2023-10-17T01:57:29.317949+0000","last_peered":"2023-10-17T01:57:29.317949+0000","last_clean":"2023-10-17T01:57:29.317949+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:57:29.317949+0000","last_undegraded":"2023-10-17T01:57:29.317949+0000","last_fullsized":"2023-10-17T01:57:29.317949+0000","mapping_epoch":16,"log_start":"101'469416","ondisk_log_start":"101'469416","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6844669971,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":479518,"num_read_kb":1887236,"num_write":478176,"num_write_kb":1912698,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"103'9535","reported_seq":13445,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-17T01:57:28.703290+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T01:57:28.703290+0000","last_peered":"2023-10-17T01:57:28.703290+0000","last_clean":"2023-10-17T01:57:28.703290+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:57:28.703290+0000","last_undegraded":"2023-10-17T01:57:28.703290+0000","last_fullsized":"2023-10-17T01:57:28.703290+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":9535,"log_dups_size":0,"ondisk_log_size":9535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":277,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-17T01:56:38.997616+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:56:38.997616+0000","last_peered":"2023-10-17T01:56:38.997616+0000","last_clean":"2023-10-17T01:56:38.997616+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:56:38.997616+0000","last_undegraded":"2023-10-17T01:56:38.997616+0000","last_fullsized":"2023-10-17T01:56:38.997616+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27530911763,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1926224,"num_read_kb":7582948,"num_write":1924102,"num_write_kb":7696402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15758536704,"data_stored":19159396390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40191,"ondisk_log_size":40191,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":45301,"ondisk_log_size":45301,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739276,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9319232,"kb_used_data":7692664,"kb_used_omap":670,"kb_used_meta":1625889,"kb_avail":85052608,"statfs":{"total":96636764160,"available":87093870592,"internally_reserved":0,"allocated":7877287936,"data_stored":9577120408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":686480,"internal_metadata":1664910960},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:57:14 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.38800000000000001,"15min":0.373},"min":{"1min":0.20499999999999999,"5min":0.189,"15min":0.16900000000000001},"max":{"1min":0.40300000000000002,"5min":3.7679999999999998,"15min":4.2140000000000004},"last":0.316},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.54400000000000004,"15min":0.47999999999999998},"min":{"1min":0.218,"5min":0.188,"15min":0.188},"max":{"1min":2.2050000000000001,"5min":5.3280000000000003,"15min":5.3280000000000003},"last":0.28899999999999998}]},{"osd":1,"last update":"Tue Oct 17 01:56:40 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.51000000000000001,"15min":0.54900000000000004},"min":{"1min":0.22600000000000001,"5min":0.20699999999999999,"15min":0.19400000000000001},"max":{"1min":9.5429999999999993,"5min":9.5429999999999993,"15min":12.413},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.59899999999999998,"15min":0.58699999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.19500000000000001},"max":{"1min":9.5999999999999996,"5min":9.5999999999999996,"15min":9.5999999999999996},"last":0.27200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739273,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5351400,"kb_used_data":3790560,"kb_used_omap":285,"kb_used_meta":1560546,"kb_avail":89020440,"statfs":{"total":96636764160,"available":91156930560,"internally_reserved":0,"allocated":3881533440,"data_stored":4719139511,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":292849,"internal_metadata":1597999119},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,3,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.51600000000000001,"15min":0.53900000000000003},"min":{"1min":0.22900000000000001,"5min":0.188,"15min":0.16200000000000001},"max":{"1min":1.3460000000000001,"5min":6.6890000000000001,"15min":23.177},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.56499999999999995,"15min":0.58999999999999997},"min":{"1min":0.28100000000000003,"5min":0.246,"15min":0.20200000000000001},"max":{"1min":1.3899999999999999,"5min":6.7539999999999996,"15min":23.207999999999998},"last":0.31900000000000001}]},{"osd":2,"last update":"Tue Oct 17 01:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.48699999999999999,"15min":0.55600000000000005},"min":{"1min":0.26400000000000001,"5min":0.188,"15min":0.184},"max":{"1min":4.194,"5min":4.2539999999999996,"15min":13.757},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.745,"5min":0.46200000000000002,"15min":0.59499999999999997},"min":{"1min":0.23999999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.0010000000000003,"5min":4.0010000000000003,"15min":14.195},"last":0.34100000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739273,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5480912,"kb_used_data":3907208,"kb_used_omap":291,"kb_used_meta":1573404,"kb_avail":88890928,"statfs":{"total":96636764160,"available":91024310272,"internally_reserved":0,"allocated":4000980992,"data_stored":4863432574,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":298083,"internal_metadata":1611166621},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,1,0,2,8,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.39100000000000001,"15min":0.48299999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.16200000000000001},"max":{"1min":0.90700000000000003,"5min":4.1680000000000001,"15min":5.9829999999999997},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.48499999999999999,"15min":0.52000000000000002},"min":{"1min":0.22500000000000001,"5min":0.191,"15min":0.189},"max":{"1min":0.71499999999999997,"5min":5.6909999999999998,"15min":6.0750000000000002},"last":0.20899999999999999}]},{"osd":2,"last update":"Tue Oct 17 01:56:46 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.38,"15min":0.35699999999999998},"min":{"1min":0.20300000000000001,"5min":0.191,"15min":0.18099999999999999},"max":{"1min":1.7230000000000001,"5min":2.6120000000000001,"15min":2.8849999999999998},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.68799999999999994,"15min":0.56100000000000005},"min":{"1min":0.23699999999999999,"5min":0.222,"15min":0.19400000000000001},"max":{"1min":1.738,"5min":10.378,"15min":10.378},"last":0.254}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4000362496,"data_stored":4862959616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3880759296,"data_stored":4718592019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7877414912,"data_stored":9577844755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:57:31.317 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:57:31.318 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:57:31.318 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:57:31.318 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:57:31.319 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T01:57:36.003 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:57:35.996+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:57:36.022 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:57:36.016+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:57:36.023 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:57:36.016+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:57:36.032 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:57:36.020+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:57:36.032 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:57:36.020+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:57:36.046 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:57:36.032+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:57:36.047 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:57:36.032+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:57:47.700 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:57:47.696+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:57:47.701 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:57:47.696+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:57:47.701 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:57:47.696+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:57:47.702 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:57:47.696+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:57:47.716 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:57:47.712+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:57:47.716 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:57:47.712+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:57:47.717 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:57:47.712+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:57:47.717 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:57:47.712+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:57:51.320 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:57:51.873 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:57:51.874 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:57:51.920 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2373,"stamp":"2023-10-17T01:57:51.449327+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27560356696,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1957136,"num_read_kb":7631341,"num_write":1931795,"num_write_kb":7726831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85833,"ondisk_log_size":85833,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20407624,"kb_used_data":15446704,"kb_used_omap":1247,"kb_used_meta":4959648,"kb_avail":262707896,"statfs":{"total":289910292480,"available":269012885504,"internally_reserved":0,"allocated":15817424896,"data_stored":19217315021,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1277412,"internal_metadata":5078680092},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,10,8,10,15,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17928192,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4404,"num_read_kb":17580,"num_write":4399,"num_write_kb":17596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001911"},"pg_stats":[{"pgid":"2.7","version":"103'12699","reported_seq":18004,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-17T01:57:49.183601+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T01:57:49.183601+0000","last_peered":"2023-10-17T01:57:49.183601+0000","last_clean":"2023-10-17T01:57:49.183601+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:57:49.183601+0000","last_undegraded":"2023-10-17T01:57:49.183601+0000","last_fullsized":"2023-10-17T01:57:49.183601+0000","mapping_epoch":13,"log_start":"62'2600","ondisk_log_start":"62'2600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"103'9568","reported_seq":14719,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-17T01:57:49.184192+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T01:57:49.184192+0000","last_peered":"2023-10-17T01:57:49.184192+0000","last_clean":"2023-10-17T01:57:49.184192+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:57:49.184192+0000","last_undegraded":"2023-10-17T01:57:49.184192+0000","last_fullsized":"2023-10-17T01:57:49.184192+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":9568,"log_dups_size":0,"ondisk_log_size":9568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"103'6416","reported_seq":9153,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-17T01:57:49.183134+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T01:57:49.183134+0000","last_peered":"2023-10-17T01:57:49.183134+0000","last_clean":"2023-10-17T01:57:49.183134+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:57:49.183134+0000","last_undegraded":"2023-10-17T01:57:49.183134+0000","last_fullsized":"2023-10-17T01:57:49.183134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6416,"log_dups_size":0,"ondisk_log_size":6416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"103'9602","reported_seq":14161,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-17T01:57:49.183301+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T01:57:49.183301+0000","last_peered":"2023-10-17T01:57:49.183301+0000","last_clean":"2023-10-17T01:57:49.183301+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:57:49.183301+0000","last_undegraded":"2023-10-17T01:57:49.183301+0000","last_fullsized":"2023-10-17T01:57:49.183301+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":9602,"log_dups_size":0,"ondisk_log_size":9602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"103'491589","reported_seq":980151,"reported_epoch":103,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:57:49.270218+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T01:57:49.270218+0000","last_peered":"2023-10-17T01:57:49.270218+0000","last_clean":"2023-10-17T01:57:49.270218+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:57:49.270218+0000","last_undegraded":"2023-10-17T01:57:49.270218+0000","last_fullsized":"2023-10-17T01:57:49.270218+0000","mapping_epoch":16,"log_start":"101'481510","ondisk_log_start":"101'481510","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1458,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 576s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":6985990144,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":489449,"num_read_kb":1926136,"num_write":489466,"num_write_kb":1957864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":257,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-17T01:57:12.217709+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:57:12.217709+0000","last_peered":"2023-10-17T01:57:12.217709+0000","last_clean":"2023-10-17T01:57:12.217709+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:57:12.217709+0000","last_undegraded":"2023-10-17T01:57:12.217709+0000","last_fullsized":"2023-10-17T01:57:12.217709+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"103'473269","reported_seq":944202,"reported_epoch":103,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:57:49.503860+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T01:57:49.503860+0000","last_peered":"2023-10-17T01:57:49.503860+0000","last_clean":"2023-10-17T01:57:49.503860+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:57:49.503860+0000","last_undegraded":"2023-10-17T01:57:49.503860+0000","last_fullsized":"2023-10-17T01:57:49.503860+0000","mapping_epoch":16,"log_start":"101'463219","ondisk_log_start":"101'463219","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1840,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 567s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6717943808,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":471737,"num_read_kb":1858356,"num_write":470985,"num_write_kb":1883940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2696,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-17T01:57:44.071105+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T01:57:44.071105+0000","last_peered":"2023-10-17T01:57:44.071105+0000","last_clean":"2023-10-17T01:57:44.071105+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:57:44.071105+0000","last_undegraded":"2023-10-17T01:57:44.071105+0000","last_fullsized":"2023-10-17T01:57:44.071105+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"103'493009","reported_seq":982816,"reported_epoch":103,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:57:49.252764+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T01:57:49.252764+0000","last_peered":"2023-10-17T01:57:49.252764+0000","last_clean":"2023-10-17T01:57:49.252764+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:57:49.252764+0000","last_undegraded":"2023-10-17T01:57:49.252764+0000","last_fullsized":"2023-10-17T01:57:49.252764+0000","mapping_epoch":16,"log_start":"101'482923","ondisk_log_start":"101'482923","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1738,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 584s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7003815936,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":490678,"num_read_kb":1931800,"num_write":490737,"num_write_kb":1962948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4046,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-17T01:57:44.202651+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T01:57:44.202651+0000","last_peered":"2023-10-17T01:57:44.202651+0000","last_clean":"2023-10-17T01:57:44.202651+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:57:44.202651+0000","last_undegraded":"2023-10-17T01:57:44.202651+0000","last_fullsized":"2023-10-17T01:57:44.202651+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"103'481250","reported_seq":961640,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-17T01:57:49.528116+0000","last_change":"2023-10-17T01:57:48.973656+0000","last_active":"2023-10-17T01:57:49.528116+0000","last_peered":"2023-10-17T01:57:49.528116+0000","last_clean":"2023-10-17T01:57:49.528116+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:57:49.528116+0000","last_undegraded":"2023-10-17T01:57:49.528116+0000","last_fullsized":"2023-10-17T01:57:49.528116+0000","mapping_epoch":16,"log_start":"101'471216","ondisk_log_start":"101'471216","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6852014099,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":481310,"num_read_kb":1894396,"num_write":479977,"num_write_kb":1919902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"103'9572","reported_seq":13495,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-17T01:57:49.183970+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T01:57:49.183970+0000","last_peered":"2023-10-17T01:57:49.183970+0000","last_clean":"2023-10-17T01:57:49.183970+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:57:49.183970+0000","last_undegraded":"2023-10-17T01:57:49.183970+0000","last_fullsized":"2023-10-17T01:57:49.183970+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":9572,"log_dups_size":0,"ondisk_log_size":9572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":277,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-17T01:56:38.997616+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:56:38.997616+0000","last_peered":"2023-10-17T01:56:38.997616+0000","last_clean":"2023-10-17T01:56:38.997616+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:56:38.997616+0000","last_undegraded":"2023-10-17T01:56:38.997616+0000","last_fullsized":"2023-10-17T01:56:38.997616+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27559763987,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1933174,"num_read_kb":7610688,"num_write":1931165,"num_write_kb":7724654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15816593408,"data_stored":19217453094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40249,"ondisk_log_size":40249,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":45485,"ondisk_log_size":45485,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739280,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9458292,"kb_used_data":7720684,"kb_used_omap":670,"kb_used_meta":1736929,"kb_avail":84913548,"statfs":{"total":96636764160,"available":86951473152,"internally_reserved":0,"allocated":7905980416,"data_stored":9605812888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":686480,"internal_metadata":1778615920},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,3,6,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:57:14 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.38800000000000001,"15min":0.373},"min":{"1min":0.20499999999999999,"5min":0.189,"15min":0.16900000000000001},"max":{"1min":0.40300000000000002,"5min":3.7679999999999998,"15min":4.2140000000000004},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.54400000000000004,"15min":0.47999999999999998},"min":{"1min":0.218,"5min":0.188,"15min":0.188},"max":{"1min":2.2050000000000001,"5min":5.3280000000000003,"15min":5.3280000000000003},"last":0.36399999999999999}]},{"osd":1,"last update":"Tue Oct 17 01:57:42 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.52100000000000002,"15min":0.52400000000000002},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.19400000000000001},"max":{"1min":0.65800000000000003,"5min":9.5429999999999993,"15min":12.413},"last":0.318},{"interface":"front","average":{"1min":0.437,"5min":0.59099999999999997,"15min":0.57599999999999996},"min":{"1min":0.27200000000000002,"5min":0.23100000000000001,"15min":0.19500000000000001},"max":{"1min":1.8120000000000001,"5min":9.5999999999999996,"15min":9.5999999999999996},"last":0.27900000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739277,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5404096,"kb_used_data":3804472,"kb_used_omap":285,"kb_used_meta":1599330,"kb_avail":88967744,"statfs":{"total":96636764160,"available":91102969856,"internally_reserved":0,"allocated":3895779328,"data_stored":4733385399,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":292849,"internal_metadata":1637713935},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,3,5,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.51600000000000001,"15min":0.53900000000000003},"min":{"1min":0.22900000000000001,"5min":0.188,"15min":0.16200000000000001},"max":{"1min":1.3460000000000001,"5min":6.6890000000000001,"15min":23.177},"last":0.186},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.56499999999999995,"15min":0.58999999999999997},"min":{"1min":0.28100000000000003,"5min":0.246,"15min":0.20200000000000001},"max":{"1min":1.3899999999999999,"5min":6.7539999999999996,"15min":23.207999999999998},"last":0.28100000000000003}]},{"osd":2,"last update":"Tue Oct 17 01:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.48699999999999999,"15min":0.55600000000000005},"min":{"1min":0.26400000000000001,"5min":0.188,"15min":0.184},"max":{"1min":4.194,"5min":4.2539999999999996,"15min":13.757},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.745,"5min":0.46200000000000002,"15min":0.59499999999999997},"min":{"1min":0.23999999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.0010000000000003,"5min":4.0010000000000003,"15min":14.195},"last":0.26200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739277,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5545236,"kb_used_data":3921548,"kb_used_omap":291,"kb_used_meta":1623388,"kb_avail":88826604,"statfs":{"total":96636764160,"available":90958442496,"internally_reserved":0,"allocated":4015665152,"data_stored":4878116734,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":298083,"internal_metadata":1662350237},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,6,0,1,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:57:43 2023","interfaces":[{"interface":"back","average":{"1min":0.29199999999999998,"5min":0.38300000000000001,"15min":0.46500000000000002},"min":{"1min":0.17999999999999999,"5min":0.17000000000000001,"15min":0.16200000000000001},"max":{"1min":0.56299999999999994,"5min":4.1680000000000001,"15min":5.9829999999999997},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.45900000000000002,"15min":0.52300000000000002},"min":{"1min":0.20899999999999999,"5min":0.191,"15min":0.189},"max":{"1min":3.1429999999999998,"5min":5.6909999999999998,"15min":6.0750000000000002},"last":0.35299999999999998}]},{"osd":2,"last update":"Tue Oct 17 01:56:46 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.38,"15min":0.35699999999999998},"min":{"1min":0.20300000000000001,"5min":0.191,"15min":0.18099999999999999},"max":{"1min":1.7230000000000001,"5min":2.6120000000000001,"15min":2.8849999999999998},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.68799999999999994,"15min":0.56100000000000005},"min":{"1min":0.23699999999999999,"5min":0.222,"15min":0.19400000000000001},"max":{"1min":1.738,"5min":10.378,"15min":10.378},"last":0.318}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4014915584,"data_stored":4877512704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3895144448,"data_stored":4732977171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7906533376,"data_stored":9606963219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:57:51.922 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:57:51.923 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:57:51.923 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:57:51.924 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:57:51.924 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T01:58:11.925 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:58:12.448 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:58:12.448 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:58:12.491 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2383,"stamp":"2023-10-17T01:58:11.453070+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27589450584,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1964611,"num_read_kb":7661133,"num_write":1938949,"num_write_kb":7755447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85849,"ondisk_log_size":85849,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20664856,"kb_used_data":15504188,"kb_used_omap":1247,"kb_used_meta":5159392,"kb_avail":262450664,"statfs":{"total":289910292480,"available":268749479936,"internally_reserved":0,"allocated":15876288512,"data_stored":19276178637,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1277412,"internal_metadata":5283217948},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,3,6,14,13,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18100224,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4633,"num_read_kb":18448,"num_write":4453,"num_write_kb":17812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002086"},"pg_stats":[{"pgid":"2.7","version":"103'12743","reported_seq":18060,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-17T01:58:08.045097+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T01:58:08.045097+0000","last_peered":"2023-10-17T01:58:08.045097+0000","last_clean":"2023-10-17T01:58:08.045097+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:58:08.045097+0000","last_undegraded":"2023-10-17T01:58:08.045097+0000","last_fullsized":"2023-10-17T01:58:08.045097+0000","mapping_epoch":13,"log_start":"63'2700","ondisk_log_start":"63'2700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"103'9601","reported_seq":14761,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-17T01:58:08.034934+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T01:58:08.034934+0000","last_peered":"2023-10-17T01:58:08.034934+0000","last_clean":"2023-10-17T01:58:08.034934+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:58:08.034934+0000","last_undegraded":"2023-10-17T01:58:08.034934+0000","last_fullsized":"2023-10-17T01:58:08.034934+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":9601,"log_dups_size":0,"ondisk_log_size":9601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"103'6438","reported_seq":9181,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-17T01:58:08.022991+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T01:58:08.022991+0000","last_peered":"2023-10-17T01:58:08.022991+0000","last_clean":"2023-10-17T01:58:08.022991+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:58:08.022991+0000","last_undegraded":"2023-10-17T01:58:08.022991+0000","last_fullsized":"2023-10-17T01:58:08.022991+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6438,"log_dups_size":0,"ondisk_log_size":6438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"103'9635","reported_seq":14203,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-17T01:58:08.034343+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T01:58:08.034343+0000","last_peered":"2023-10-17T01:58:08.034343+0000","last_clean":"2023-10-17T01:58:08.034343+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:58:08.034343+0000","last_undegraded":"2023-10-17T01:58:08.034343+0000","last_fullsized":"2023-10-17T01:58:08.034343+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":9635,"log_dups_size":0,"ondisk_log_size":9635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"103'493418","reported_seq":983981,"reported_epoch":103,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:58:09.403566+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T01:58:09.403566+0000","last_peered":"2023-10-17T01:58:09.403566+0000","last_clean":"2023-10-17T01:58:09.403566+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:58:09.403566+0000","last_undegraded":"2023-10-17T01:58:09.403566+0000","last_fullsized":"2023-10-17T01:58:09.403566+0000","mapping_epoch":16,"log_start":"102'483410","ondisk_log_start":"102'483410","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1520,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 596s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":6993256448,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":491441,"num_read_kb":1934080,"num_write":491259,"num_write_kb":1965036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":257,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-17T01:57:12.217709+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:57:12.217709+0000","last_peered":"2023-10-17T01:57:12.217709+0000","last_clean":"2023-10-17T01:57:12.217709+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:57:12.217709+0000","last_undegraded":"2023-10-17T01:57:12.217709+0000","last_fullsized":"2023-10-17T01:57:12.217709+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"103'475018","reported_seq":947757,"reported_epoch":103,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:58:09.621028+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T01:58:09.621028+0000","last_peered":"2023-10-17T01:58:09.621028+0000","last_clean":"2023-10-17T01:58:09.621028+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:58:09.621028+0000","last_undegraded":"2023-10-17T01:58:09.621028+0000","last_fullsized":"2023-10-17T01:58:09.621028+0000","mapping_epoch":16,"log_start":"102'464919","ondisk_log_start":"102'464919","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1884,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 588s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6724968448,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":473534,"num_read_kb":1865512,"num_write":472710,"num_write_kb":1890840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2704,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-17T01:58:04.565136+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T01:58:04.565136+0000","last_peered":"2023-10-17T01:58:04.565136+0000","last_clean":"2023-10-17T01:58:04.565136+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:58:04.565136+0000","last_undegraded":"2023-10-17T01:58:04.565136+0000","last_fullsized":"2023-10-17T01:58:04.565136+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"103'494870","reported_seq":986556,"reported_epoch":103,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:58:09.385533+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T01:58:09.385533+0000","last_peered":"2023-10-17T01:58:09.385533+0000","last_clean":"2023-10-17T01:58:09.385533+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:58:09.385533+0000","last_undegraded":"2023-10-17T01:58:09.385533+0000","last_fullsized":"2023-10-17T01:58:09.385533+0000","mapping_epoch":16,"log_start":"102'484823","ondisk_log_start":"102'484823","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1804,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 604s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7011213312,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":492547,"num_read_kb":1939252,"num_write":492561,"num_write_kb":1970244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4058,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-17T01:58:04.549496+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T01:58:04.549496+0000","last_peered":"2023-10-17T01:58:04.549496+0000","last_clean":"2023-10-17T01:58:04.549496+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:58:04.549496+0000","last_undegraded":"2023-10-17T01:58:04.549496+0000","last_fullsized":"2023-10-17T01:58:04.549496+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"103'483062","reported_seq":965273,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-17T01:58:09.608135+0000","last_change":"2023-10-17T01:58:06.006758+0000","last_active":"2023-10-17T01:58:09.608135+0000","last_peered":"2023-10-17T01:58:09.608135+0000","last_clean":"2023-10-17T01:58:09.608135+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:58:09.608135+0000","last_undegraded":"2023-10-17T01:58:09.608135+0000","last_fullsized":"2023-10-17T01:58:09.608135+0000","mapping_epoch":16,"log_start":"102'473016","ondisk_log_start":"102'473016","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6859419667,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":483127,"num_read_kb":1901636,"num_write":481789,"num_write_kb":1927150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"103'9605","reported_seq":13537,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-17T01:58:08.050493+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T01:58:08.050493+0000","last_peered":"2023-10-17T01:58:08.050493+0000","last_clean":"2023-10-17T01:58:08.050493+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:58:08.050493+0000","last_undegraded":"2023-10-17T01:58:08.050493+0000","last_fullsized":"2023-10-17T01:58:08.050493+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":9605,"log_dups_size":0,"ondisk_log_size":9605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":277,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-17T01:56:38.997616+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:56:38.997616+0000","last_peered":"2023-10-17T01:56:38.997616+0000","last_clean":"2023-10-17T01:56:38.997616+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:56:38.997616+0000","last_undegraded":"2023-10-17T01:56:38.997616+0000","last_fullsized":"2023-10-17T01:56:38.997616+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27588857875,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1940649,"num_read_kb":7640480,"num_write":1938319,"num_write_kb":7753270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15874990080,"data_stored":19275849766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40200,"ondisk_log_size":40200,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":45550,"ondisk_log_size":45550,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739284,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9596576,"kb_used_data":7749464,"kb_used_omap":670,"kb_used_meta":1846433,"kb_avail":84775264,"statfs":{"total":96636764160,"available":86809870336,"internally_reserved":0,"allocated":7935451136,"data_stored":9635283608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":686480,"internal_metadata":1890748016},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:57:14 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.38800000000000001,"15min":0.373},"min":{"1min":0.20499999999999999,"5min":0.189,"15min":0.16900000000000001},"max":{"1min":0.40300000000000002,"5min":3.7679999999999998,"15min":4.2140000000000004},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.54400000000000004,"15min":0.47999999999999998},"min":{"1min":0.218,"5min":0.188,"15min":0.188},"max":{"1min":2.2050000000000001,"5min":5.3280000000000003,"15min":5.3280000000000003},"last":0.25900000000000001}]},{"osd":1,"last update":"Tue Oct 17 01:57:42 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.52100000000000002,"15min":0.52400000000000002},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.19400000000000001},"max":{"1min":0.65800000000000003,"5min":9.5429999999999993,"15min":12.413},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.437,"5min":0.59099999999999997,"15min":0.57599999999999996},"min":{"1min":0.27200000000000002,"5min":0.23100000000000001,"15min":0.19500000000000001},"max":{"1min":1.8120000000000001,"5min":9.5999999999999996,"15min":9.5999999999999996},"last":0.38}]}]},{"osd":1,"up_from":8,"seq":34359739281,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5472452,"kb_used_data":3818684,"kb_used_omap":285,"kb_used_meta":1653474,"kb_avail":88899388,"statfs":{"total":96636764160,"available":91032973312,"internally_reserved":0,"allocated":3910332416,"data_stored":4747938487,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":292849,"internal_metadata":1693157391},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:58:05 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.501,"15min":0.53900000000000003},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16200000000000001},"max":{"1min":0.441,"5min":6.6890000000000001,"15min":23.177},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55300000000000005,"15min":0.58999999999999997},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.59999999999999998,"5min":6.7539999999999996,"15min":23.207999999999998},"last":0.38900000000000001}]},{"osd":2,"last update":"Tue Oct 17 01:58:05 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.503,"15min":0.56200000000000006},"min":{"1min":0.19700000000000001,"5min":0.192,"15min":0.184},"max":{"1min":2.0209999999999999,"5min":4.2539999999999996,"15min":13.757},"last":0.249},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.443,"15min":0.59099999999999997},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.46300000000000002,"5min":4.0010000000000003,"15min":14.195},"last":0.42499999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739281,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5595828,"kb_used_data":3936040,"kb_used_omap":291,"kb_used_meta":1659484,"kb_avail":88776012,"statfs":{"total":96636764160,"available":90906636288,"internally_reserved":0,"allocated":4030504960,"data_stored":4892956542,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":298083,"internal_metadata":1699312541},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,6,9,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:57:43 2023","interfaces":[{"interface":"back","average":{"1min":0.29199999999999998,"5min":0.38300000000000001,"15min":0.46500000000000002},"min":{"1min":0.17999999999999999,"5min":0.17000000000000001,"15min":0.16200000000000001},"max":{"1min":0.56299999999999994,"5min":4.1680000000000001,"15min":5.9829999999999997},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.45900000000000002,"15min":0.52300000000000002},"min":{"1min":0.20899999999999999,"5min":0.191,"15min":0.189},"max":{"1min":3.1429999999999998,"5min":5.6909999999999998,"15min":6.0750000000000002},"last":0.40000000000000002}]},{"osd":2,"last update":"Tue Oct 17 01:57:50 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.39900000000000002,"15min":0.35799999999999998},"min":{"1min":0.23699999999999999,"5min":0.191,"15min":0.18099999999999999},"max":{"1min":2.5339999999999998,"5min":2.6120000000000001,"15min":2.6120000000000001},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.65400000000000003,"15min":0.57699999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.19400000000000001},"max":{"1min":8.827,"5min":10.378,"15min":10.378},"last":0.317}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4029734912,"data_stored":4892332032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3909623808,"data_stored":4747456531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7935631360,"data_stored":9636061203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:58:12.493 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:58:12.493 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:58:12.494 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:58:12.494 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:58:12.494 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T01:58:32.496 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:58:33.049 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:58:33.051 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:58:33.088 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2394,"stamp":"2023-10-17T01:58:31.456544+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27618216792,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1971838,"num_read_kb":7689905,"num_write":1946034,"num_write_kb":7783787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86009,"ondisk_log_size":86009,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20928088,"kb_used_data":15562172,"kb_used_omap":1247,"kb_used_meta":5364640,"kb_avail":262187432,"statfs":{"total":289910292480,"available":268479930368,"internally_reserved":0,"allocated":15935664128,"data_stored":19335541695,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1277412,"internal_metadata":5493391900},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,2,9,13,17,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21622784,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5371,"num_read_kb":21388,"num_write":5314,"num_write_kb":21256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002081"},"pg_stats":[{"pgid":"2.7","version":"104'12791","reported_seq":18126,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-10-17T01:58:29.329359+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T01:58:29.329359+0000","last_peered":"2023-10-17T01:58:29.329359+0000","last_clean":"2023-10-17T01:58:29.329359+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:58:29.329359+0000","last_undegraded":"2023-10-17T01:58:29.329359+0000","last_fullsized":"2023-10-17T01:58:29.329359+0000","mapping_epoch":13,"log_start":"63'2700","ondisk_log_start":"63'2700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"104'9637","reported_seq":14811,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-10-17T01:58:29.322889+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T01:58:29.322889+0000","last_peered":"2023-10-17T01:58:29.322889+0000","last_clean":"2023-10-17T01:58:29.322889+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:58:29.322889+0000","last_undegraded":"2023-10-17T01:58:29.322889+0000","last_fullsized":"2023-10-17T01:58:29.322889+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":9637,"log_dups_size":0,"ondisk_log_size":9637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"104'6462","reported_seq":9214,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-10-17T01:58:29.310759+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T01:58:29.310759+0000","last_peered":"2023-10-17T01:58:29.310759+0000","last_clean":"2023-10-17T01:58:29.310759+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:58:29.310759+0000","last_undegraded":"2023-10-17T01:58:29.310759+0000","last_fullsized":"2023-10-17T01:58:29.310759+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6462,"log_dups_size":0,"ondisk_log_size":6462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"104'9671","reported_seq":14252,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-10-17T01:58:29.322062+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T01:58:29.322062+0000","last_peered":"2023-10-17T01:58:29.322062+0000","last_clean":"2023-10-17T01:58:29.322062+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:58:29.322062+0000","last_undegraded":"2023-10-17T01:58:29.322062+0000","last_fullsized":"2023-10-17T01:58:29.322062+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":9671,"log_dups_size":0,"ondisk_log_size":9671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"104'495232","reported_seq":987623,"reported_epoch":104,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:58:29.544701+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T01:58:29.544701+0000","last_peered":"2023-10-17T01:58:29.544701+0000","last_clean":"2023-10-17T01:58:29.544701+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:58:29.544701+0000","last_undegraded":"2023-10-17T01:58:29.544701+0000","last_fullsized":"2023-10-17T01:58:29.544701+0000","mapping_epoch":16,"log_start":"102'485210","ondisk_log_start":"102'485210","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1564,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 616s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7000539136,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":493263,"num_read_kb":1941336,"num_write":493050,"num_write_kb":1972200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":258,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-17T01:57:39.209747+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:57:39.209747+0000","last_peered":"2023-10-17T01:57:39.209747+0000","last_clean":"2023-10-17T01:57:39.209747+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:57:39.209747+0000","last_undegraded":"2023-10-17T01:57:39.209747+0000","last_fullsized":"2023-10-17T01:57:39.209747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"104'476798","reported_seq":951298,"reported_epoch":104,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:58:29.755925+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T01:58:29.755925+0000","last_peered":"2023-10-17T01:58:29.755925+0000","last_clean":"2023-10-17T01:58:29.755925+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:58:29.755925+0000","last_undegraded":"2023-10-17T01:58:29.755925+0000","last_fullsized":"2023-10-17T01:58:29.755925+0000","mapping_epoch":16,"log_start":"102'466719","ondisk_log_start":"102'466719","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1938,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 608s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6732083200,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":475282,"num_read_kb":1872460,"num_write":474461,"num_write_kb":1897844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2713,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-10-17T01:58:25.023196+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T01:58:25.023196+0000","last_peered":"2023-10-17T01:58:25.023196+0000","last_clean":"2023-10-17T01:58:25.023196+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:58:25.023196+0000","last_undegraded":"2023-10-17T01:58:25.023196+0000","last_fullsized":"2023-10-17T01:58:25.023196+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"104'496686","reported_seq":990171,"reported_epoch":104,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:58:29.536370+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T01:58:29.536370+0000","last_peered":"2023-10-17T01:58:29.536370+0000","last_clean":"2023-10-17T01:58:29.536370+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:58:29.536370+0000","last_undegraded":"2023-10-17T01:58:29.536370+0000","last_fullsized":"2023-10-17T01:58:29.536370+0000","mapping_epoch":16,"log_start":"102'486623","ondisk_log_start":"102'486623","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1876,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 624s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7018414080,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":494333,"num_read_kb":1946380,"num_write":494334,"num_write_kb":1977336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4071,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-10-17T01:58:25.034732+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T01:58:25.034732+0000","last_peered":"2023-10-17T01:58:25.034732+0000","last_clean":"2023-10-17T01:58:25.034732+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:58:25.034732+0000","last_undegraded":"2023-10-17T01:58:25.034732+0000","last_fullsized":"2023-10-17T01:58:25.034732+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"104'484832","reported_seq":968923,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-10-17T01:58:29.783530+0000","last_change":"2023-10-17T01:58:28.311217+0000","last_active":"2023-10-17T01:58:29.783530+0000","last_peered":"2023-10-17T01:58:29.783530+0000","last_clean":"2023-10-17T01:58:29.783530+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:58:29.783530+0000","last_undegraded":"2023-10-17T01:58:29.783530+0000","last_fullsized":"2023-10-17T01:58:29.783530+0000","mapping_epoch":16,"log_start":"102'474816","ondisk_log_start":"102'474816","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6866587667,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":484998,"num_read_kb":1909076,"num_write":483559,"num_write_kb":1934230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"104'9641","reported_seq":13586,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-10-17T01:58:29.337363+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T01:58:29.337363+0000","last_peered":"2023-10-17T01:58:29.337363+0000","last_clean":"2023-10-17T01:58:29.337363+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:58:29.337363+0000","last_undegraded":"2023-10-17T01:58:29.337363+0000","last_fullsized":"2023-10-17T01:58:29.337363+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":9641,"log_dups_size":0,"ondisk_log_size":9641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":278,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-17T01:57:12.204676+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:57:12.204676+0000","last_peered":"2023-10-17T01:57:12.204676+0000","last_clean":"2023-10-17T01:57:12.204676+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:57:12.204676+0000","last_undegraded":"2023-10-17T01:57:12.204676+0000","last_fullsized":"2023-10-17T01:57:12.204676+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27617624083,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1947876,"num_read_kb":7669252,"num_write":1945404,"num_write_kb":7781610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15933267968,"data_stored":19334127654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40180,"ondisk_log_size":40180,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":45730,"ondisk_log_size":45730,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739288,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9759260,"kb_used_data":7778964,"kb_used_omap":670,"kb_used_meta":1979617,"kb_avail":84612580,"statfs":{"total":96636764160,"available":86643281920,"internally_reserved":0,"allocated":7965659136,"data_stored":9665487422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":686480,"internal_metadata":2027128432},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,5,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:58:17 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.36399999999999999,"15min":0.379},"min":{"1min":0.20200000000000001,"5min":0.189,"15min":0.16900000000000001},"max":{"1min":2.1600000000000001,"5min":2.5790000000000002,"15min":4.2140000000000004},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.50700000000000001,"15min":0.47299999999999998},"min":{"1min":0.23300000000000001,"5min":0.19800000000000001,"15min":0.188},"max":{"1min":2.194,"5min":5.3280000000000003,"15min":5.3280000000000003},"last":0.505}]},{"osd":1,"last update":"Tue Oct 17 01:57:42 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.52100000000000002,"15min":0.52400000000000002},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.19400000000000001},"max":{"1min":0.65800000000000003,"5min":9.5429999999999993,"15min":12.413},"last":0.434},{"interface":"front","average":{"1min":0.437,"5min":0.59099999999999997,"15min":0.57599999999999996},"min":{"1min":0.27200000000000002,"5min":0.23100000000000001,"15min":0.19500000000000001},"max":{"1min":1.8120000000000001,"5min":9.5999999999999996,"15min":9.5999999999999996},"last":0.46600000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739285,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5518588,"kb_used_data":3832880,"kb_used_omap":285,"kb_used_meta":1685410,"kb_avail":88853252,"statfs":{"total":96636764160,"available":90985730048,"internally_reserved":0,"allocated":3924869120,"data_stored":4762471005,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":292849,"internal_metadata":1725859855},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,6,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:58:05 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.501,"15min":0.53900000000000003},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16200000000000001},"max":{"1min":0.441,"5min":6.6890000000000001,"15min":23.177},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55300000000000005,"15min":0.58999999999999997},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.59999999999999998,"5min":6.7539999999999996,"15min":23.207999999999998},"last":0.38900000000000001}]},{"osd":2,"last update":"Tue Oct 17 01:58:05 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.503,"15min":0.56200000000000006},"min":{"1min":0.19700000000000001,"5min":0.192,"15min":0.184},"max":{"1min":2.0209999999999999,"5min":4.2539999999999996,"15min":13.757},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.443,"15min":0.59099999999999997},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.46300000000000002,"5min":4.0010000000000003,"15min":14.195},"last":0.33000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739285,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5650240,"kb_used_data":3950328,"kb_used_omap":291,"kb_used_meta":1699612,"kb_avail":88721600,"statfs":{"total":96636764160,"available":90850918400,"internally_reserved":0,"allocated":4045135872,"data_stored":4907583268,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":298083,"internal_metadata":1740403613},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,0,0,2,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:57:43 2023","interfaces":[{"interface":"back","average":{"1min":0.29199999999999998,"5min":0.38300000000000001,"15min":0.46500000000000002},"min":{"1min":0.17999999999999999,"5min":0.17000000000000001,"15min":0.16200000000000001},"max":{"1min":0.56299999999999994,"5min":4.1680000000000001,"15min":5.9829999999999997},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.45900000000000002,"15min":0.52300000000000002},"min":{"1min":0.20899999999999999,"5min":0.191,"15min":0.189},"max":{"1min":3.1429999999999998,"5min":5.6909999999999998,"15min":6.0750000000000002},"last":0.22700000000000001}]},{"osd":2,"last update":"Tue Oct 17 01:57:50 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.39900000000000002,"15min":0.35799999999999998},"min":{"1min":0.23699999999999999,"5min":0.191,"15min":0.18099999999999999},"max":{"1min":2.5339999999999998,"5min":2.6120000000000001,"15min":2.6120000000000001},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.65400000000000003,"15min":0.57699999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.19400000000000001},"max":{"1min":8.827,"5min":10.378,"15min":10.378},"last":0.28699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4044341248,"data_stored":4906938368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3924054016,"data_stored":4761886739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7964872704,"data_stored":9665302547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:58:33.090 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:58:33.091 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:58:33.091 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:58:33.092 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:58:33.092 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T01:58:45.686 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:58:45.679+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:58:45.686 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:58:45.679+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:58:45.697 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:58:45.691+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:58:45.703 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:58:45.695+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:58:45.704 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:58:45.695+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:58:45.704 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:58:45.695+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:58:45.704 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:58:45.695+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:58:53.093 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:58:53.714 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:58:53.714 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:58:53.765 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2405,"stamp":"2023-10-17T01:58:53.460324+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27646958424,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1978754,"num_read_kb":7717409,"num_write":1953095,"num_write_kb":7812031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86240,"ondisk_log_size":86240,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21186236,"kb_used_data":15625956,"kb_used_omap":1247,"kb_used_meta":5559008,"kb_avail":261929284,"statfs":{"total":289910292480,"available":268215586816,"internally_reserved":0,"allocated":16000978944,"data_stored":19400856511,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1277412,"internal_metadata":5692424732},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,2,5,3,14,20,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14303232,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3505,"num_read_kb":13932,"num_write":3518,"num_write_kb":14072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002064"},"pg_stats":[{"pgid":"2.7","version":"104'12868","reported_seq":18219,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-10-17T01:58:48.171002+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T01:58:48.171002+0000","last_peered":"2023-10-17T01:58:48.171002+0000","last_clean":"2023-10-17T01:58:48.171002+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:58:48.171002+0000","last_undegraded":"2023-10-17T01:58:48.171002+0000","last_fullsized":"2023-10-17T01:58:48.171002+0000","mapping_epoch":13,"log_start":"67'2800","ondisk_log_start":"67'2800","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"104'9696","reported_seq":14882,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-10-17T01:58:48.146671+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T01:58:48.146671+0000","last_peered":"2023-10-17T01:58:48.146671+0000","last_clean":"2023-10-17T01:58:48.146671+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:58:48.146671+0000","last_undegraded":"2023-10-17T01:58:48.146671+0000","last_fullsized":"2023-10-17T01:58:48.146671+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":9696,"log_dups_size":0,"ondisk_log_size":9696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"104'6502","reported_seq":9262,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-10-17T01:58:48.142871+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T01:58:48.142871+0000","last_peered":"2023-10-17T01:58:48.142871+0000","last_clean":"2023-10-17T01:58:48.142871+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:58:48.142871+0000","last_undegraded":"2023-10-17T01:58:48.142871+0000","last_fullsized":"2023-10-17T01:58:48.142871+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6502,"log_dups_size":0,"ondisk_log_size":6502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"104'9729","reported_seq":14322,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-10-17T01:58:48.145556+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T01:58:48.145556+0000","last_peered":"2023-10-17T01:58:48.145556+0000","last_clean":"2023-10-17T01:58:48.145556+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:58:48.145556+0000","last_undegraded":"2023-10-17T01:58:48.145556+0000","last_fullsized":"2023-10-17T01:58:48.145556+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":9729,"log_dups_size":0,"ondisk_log_size":9729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"104'497074","reported_seq":991211,"reported_epoch":104,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:58:49.668761+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T01:58:49.668761+0000","last_peered":"2023-10-17T01:58:49.668761+0000","last_clean":"2023-10-17T01:58:49.668761+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:58:49.668761+0000","last_undegraded":"2023-10-17T01:58:49.668761+0000","last_fullsized":"2023-10-17T01:58:49.668761+0000","mapping_epoch":16,"log_start":"102'487010","ondisk_log_start":"102'487010","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1610,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 636s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7007936512,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":494998,"num_read_kb":1948216,"num_write":494867,"num_write_kb":1979468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":258,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-17T01:57:39.209747+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:57:39.209747+0000","last_peered":"2023-10-17T01:57:39.209747+0000","last_clean":"2023-10-17T01:57:39.209747+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:57:39.209747+0000","last_undegraded":"2023-10-17T01:57:39.209747+0000","last_fullsized":"2023-10-17T01:57:39.209747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"104'478527","reported_seq":954725,"reported_epoch":104,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:58:49.899857+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T01:58:49.899857+0000","last_peered":"2023-10-17T01:58:49.899857+0000","last_clean":"2023-10-17T01:58:49.899857+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:58:49.899857+0000","last_undegraded":"2023-10-17T01:58:49.899857+0000","last_fullsized":"2023-10-17T01:58:49.899857+0000","mapping_epoch":16,"log_start":"102'468519","ondisk_log_start":"102'468519","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1988,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 628s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6738997248,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":476970,"num_read_kb":1879172,"num_write":476162,"num_write_kb":1904648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2723,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-10-17T01:58:50.627063+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T01:58:50.627063+0000","last_peered":"2023-10-17T01:58:50.627063+0000","last_clean":"2023-10-17T01:58:50.627063+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:58:50.627063+0000","last_undegraded":"2023-10-17T01:58:50.627063+0000","last_fullsized":"2023-10-17T01:58:50.627063+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"104'498473","reported_seq":993718,"reported_epoch":104,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:58:49.654079+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T01:58:49.654079+0000","last_peered":"2023-10-17T01:58:49.654079+0000","last_clean":"2023-10-17T01:58:49.654079+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:58:49.654079+0000","last_undegraded":"2023-10-17T01:58:49.654079+0000","last_fullsized":"2023-10-17T01:58:49.654079+0000","mapping_epoch":16,"log_start":"102'488423","ondisk_log_start":"102'488423","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1920,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 644s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7025577984,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":496085,"num_read_kb":1953356,"num_write":496097,"num_write_kb":1984388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4087,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-10-17T01:58:50.624054+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T01:58:50.624054+0000","last_peered":"2023-10-17T01:58:50.624054+0000","last_clean":"2023-10-17T01:58:50.624054+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:58:50.624054+0000","last_undegraded":"2023-10-17T01:58:50.624054+0000","last_fullsized":"2023-10-17T01:58:50.624054+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"104'486612","reported_seq":972454,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-10-17T01:58:49.918749+0000","last_change":"2023-10-17T01:58:49.599130+0000","last_active":"2023-10-17T01:58:49.918749+0000","last_peered":"2023-10-17T01:58:49.918749+0000","last_clean":"2023-10-17T01:58:49.918749+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:58:49.918749+0000","last_undegraded":"2023-10-17T01:58:49.918749+0000","last_fullsized":"2023-10-17T01:58:49.918749+0000","mapping_epoch":16,"log_start":"102'476516","ondisk_log_start":"102'476516","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6873853971,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":486739,"num_read_kb":1916012,"num_write":485339,"num_write_kb":1941350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"104'9700","reported_seq":13657,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-10-17T01:58:48.195983+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T01:58:48.195983+0000","last_peered":"2023-10-17T01:58:48.195983+0000","last_clean":"2023-10-17T01:58:48.195983+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:58:48.195983+0000","last_undegraded":"2023-10-17T01:58:48.195983+0000","last_fullsized":"2023-10-17T01:58:48.195983+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":9700,"log_dups_size":0,"ondisk_log_size":9700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":279,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-10-17T01:58:17.244435+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:58:17.244435+0000","last_peered":"2023-10-17T01:58:17.244435+0000","last_clean":"2023-10-17T01:58:17.244435+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:58:17.244435+0000","last_undegraded":"2023-10-17T01:58:17.244435+0000","last_fullsized":"2023-10-17T01:58:17.244435+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27646365715,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1954792,"num_read_kb":7696756,"num_write":1952465,"num_write_kb":7809854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15998435328,"data_stored":19399295014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40218,"ondisk_log_size":40218,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":45923,"ondisk_log_size":45923,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739293,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9902684,"kb_used_data":7814100,"kb_used_omap":670,"kb_used_meta":2087905,"kb_avail":84469156,"statfs":{"total":96636764160,"available":86496415744,"internally_reserved":0,"allocated":8001638400,"data_stored":9701466686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":686480,"internal_metadata":2138015344},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,2,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:58:17 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.36399999999999999,"15min":0.379},"min":{"1min":0.20200000000000001,"5min":0.189,"15min":0.16900000000000001},"max":{"1min":2.1600000000000001,"5min":2.5790000000000002,"15min":4.2140000000000004},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.50700000000000001,"15min":0.47299999999999998},"min":{"1min":0.23300000000000001,"5min":0.19800000000000001,"15min":0.188},"max":{"1min":2.194,"5min":5.3280000000000003,"15min":5.3280000000000003},"last":0.26900000000000002}]},{"osd":1,"last update":"Tue Oct 17 01:58:42 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.54100000000000004,"15min":0.51300000000000001},"min":{"1min":0.252,"5min":0.214,"15min":0.19400000000000001},"max":{"1min":5.2169999999999996,"5min":9.5429999999999993,"15min":9.5429999999999993},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.65800000000000003,"15min":0.58599999999999997},"min":{"1min":0.27900000000000003,"5min":0.23100000000000001,"15min":0.19500000000000001},"max":{"1min":5.4379999999999997,"5min":9.5999999999999996,"15min":9.5999999999999996},"last":0.39600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739289,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5568764,"kb_used_data":3846964,"kb_used_omap":285,"kb_used_meta":1721506,"kb_avail":88803076,"statfs":{"total":96636764160,"available":90934349824,"internally_reserved":0,"allocated":3939291136,"data_stored":4776893021,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":292849,"internal_metadata":1762822159},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,0,2,4,13,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:58:05 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.501,"15min":0.53900000000000003},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16200000000000001},"max":{"1min":0.441,"5min":6.6890000000000001,"15min":23.177},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55300000000000005,"15min":0.58999999999999997},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.59999999999999998,"5min":6.7539999999999996,"15min":23.207999999999998},"last":0.34799999999999998}]},{"osd":2,"last update":"Tue Oct 17 01:58:05 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.503,"15min":0.56200000000000006},"min":{"1min":0.19700000000000001,"5min":0.192,"15min":0.184},"max":{"1min":2.0209999999999999,"5min":4.2539999999999996,"15min":13.757},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.443,"15min":0.59099999999999997},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.46300000000000002,"5min":4.0010000000000003,"15min":14.195},"last":0.24399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739289,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5714788,"kb_used_data":3964892,"kb_used_omap":291,"kb_used_meta":1749596,"kb_avail":88657052,"statfs":{"total":96636764160,"available":90784821248,"internally_reserved":0,"allocated":4060049408,"data_stored":4922496804,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":298083,"internal_metadata":1791587229},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,1,8,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40400000000000003,"15min":0.44800000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.16200000000000001},"max":{"1min":2.02,"5min":4.1680000000000001,"15min":5.9829999999999997},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.41299999999999998,"15min":0.505},"min":{"1min":0.191,"5min":0.191,"15min":0.189},"max":{"1min":0.81599999999999995,"5min":3.1429999999999998,"15min":6.0750000000000002},"last":0.23200000000000001}]},{"osd":2,"last update":"Tue Oct 17 01:57:50 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.39900000000000002,"15min":0.35799999999999998},"min":{"1min":0.23699999999999999,"5min":0.191,"15min":0.18099999999999999},"max":{"1min":2.5339999999999998,"5min":2.6120000000000001,"15min":2.6120000000000001},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.65400000000000003,"15min":0.57699999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.19400000000000001},"max":{"1min":8.827,"5min":10.378,"15min":10.378},"last":0.26700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4059004928,"data_stored":4921602048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3938308096,"data_stored":4776140819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8001122304,"data_stored":9701552147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:58:53.767 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:58:53.768 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:58:53.768 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:58:53.769 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:58:53.769 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-17T01:58:54.381 INFO:teuthology.orchestra.run.smithi169.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-17T01:58:54.445 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-17T01:58:55.028 INFO:teuthology.orchestra.run.smithi169.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-17T01:58:55.059 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-17T01:58:55.694 INFO:teuthology.orchestra.run.smithi169.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-17T01:58:55.723 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-17T01:58:56.246 INFO:teuthology.orchestra.run.smithi169.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-17T01:58:56.274 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T01:59:10.072 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:59:10.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T01:59:10.073 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:59:10.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T01:59:10.073 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:59:10.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbda750 2023-10-17T01:59:10.073 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:59:10.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T01:59:10.073 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:59:10.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T01:59:10.074 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:59:10.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T01:59:10.074 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:59:10.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T01:59:10.074 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:59:10.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:59:10.075 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:59:10.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T01:59:10.075 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:59:10.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:59:10.076 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:59:10.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T01:59:10.076 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:59:10.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T01:59:10.076 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:59:10.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T01:59:10.077 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:59:10.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T01:59:10.077 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:59:10.066+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T01:59:16.275 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:59:16.807 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:59:16.809 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:59:16.856 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2416,"stamp":"2023-10-17T01:59:15.464259+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27682823000,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1987182,"num_read_kb":7750893,"num_write":1961951,"num_write_kb":7847455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86244,"ondisk_log_size":86244,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21102520,"kb_used_data":15688416,"kb_used_omap":1247,"kb_used_meta":5412832,"kb_avail":262013000,"statfs":{"total":289910292480,"available":268301312000,"internally_reserved":0,"allocated":16064937984,"data_stored":19464815551,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1277412,"internal_metadata":5542740508},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,6,10,23,7,9,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21504000,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5081,"num_read_kb":20192,"num_write":5317,"num_write_kb":21268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002172"},"pg_stats":[{"pgid":"2.7","version":"104'12908","reported_seq":18279,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-10-17T01:59:14.989406+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T01:59:14.989406+0000","last_peered":"2023-10-17T01:59:14.989406+0000","last_clean":"2023-10-17T01:59:14.989406+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:59:14.989406+0000","last_undegraded":"2023-10-17T01:59:14.989406+0000","last_fullsized":"2023-10-17T01:59:14.989406+0000","mapping_epoch":13,"log_start":"67'2900","ondisk_log_start":"67'2900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"104'9726","reported_seq":14927,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-10-17T01:59:14.988136+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T01:59:14.988136+0000","last_peered":"2023-10-17T01:59:14.988136+0000","last_clean":"2023-10-17T01:59:14.988136+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:59:14.988136+0000","last_undegraded":"2023-10-17T01:59:14.988136+0000","last_fullsized":"2023-10-17T01:59:14.988136+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":9726,"log_dups_size":0,"ondisk_log_size":9726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"104'6522","reported_seq":9292,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-10-17T01:59:14.970092+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T01:59:14.970092+0000","last_peered":"2023-10-17T01:59:14.970092+0000","last_clean":"2023-10-17T01:59:14.970092+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:59:14.970092+0000","last_undegraded":"2023-10-17T01:59:14.970092+0000","last_fullsized":"2023-10-17T01:59:14.970092+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6522,"log_dups_size":0,"ondisk_log_size":6522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"104'9759","reported_seq":14367,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-10-17T01:59:14.987191+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T01:59:14.987191+0000","last_peered":"2023-10-17T01:59:14.987191+0000","last_clean":"2023-10-17T01:59:14.987191+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:59:14.987191+0000","last_undegraded":"2023-10-17T01:59:14.987191+0000","last_fullsized":"2023-10-17T01:59:14.987191+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":9759,"log_dups_size":0,"ondisk_log_size":9759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"104'499330","reported_seq":995650,"reported_epoch":104,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:59:14.866589+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T01:59:14.866589+0000","last_peered":"2023-10-17T01:59:14.866589+0000","last_clean":"2023-10-17T01:59:14.866589+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:59:14.866589+0000","last_undegraded":"2023-10-17T01:59:14.866589+0000","last_fullsized":"2023-10-17T01:59:14.866589+0000","mapping_epoch":16,"log_start":"103'489310","ondisk_log_start":"103'489310","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1652,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 661s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7016988672,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":497167,"num_read_kb":1956840,"num_write":497100,"num_write_kb":1988400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":258,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-17T01:57:39.209747+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:57:39.209747+0000","last_peered":"2023-10-17T01:57:39.209747+0000","last_clean":"2023-10-17T01:57:39.209747+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:57:39.209747+0000","last_undegraded":"2023-10-17T01:57:39.209747+0000","last_fullsized":"2023-10-17T01:57:39.209747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"104'480666","reported_seq":958894,"reported_epoch":104,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:59:15.084785+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T01:59:15.084785+0000","last_peered":"2023-10-17T01:59:15.084785+0000","last_clean":"2023-10-17T01:59:15.084785+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:59:15.084785+0000","last_undegraded":"2023-10-17T01:59:15.084785+0000","last_fullsized":"2023-10-17T01:59:15.084785+0000","mapping_epoch":16,"log_start":"103'470619","ondisk_log_start":"103'470619","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2070,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 653s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6747459584,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":478987,"num_read_kb":1887184,"num_write":478255,"num_write_kb":1913020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2731,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-10-17T01:59:11.141910+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T01:59:11.141910+0000","last_peered":"2023-10-17T01:59:11.141910+0000","last_clean":"2023-10-17T01:59:11.141910+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:59:11.141910+0000","last_undegraded":"2023-10-17T01:59:11.141910+0000","last_fullsized":"2023-10-17T01:59:11.141910+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"104'500800","reported_seq":998171,"reported_epoch":104,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:59:14.861292+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T01:59:14.861292+0000","last_peered":"2023-10-17T01:59:14.861292+0000","last_clean":"2023-10-17T01:59:14.861292+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:59:14.861292+0000","last_undegraded":"2023-10-17T01:59:14.861292+0000","last_fullsized":"2023-10-17T01:59:14.861292+0000","mapping_epoch":16,"log_start":"103'490723","ondisk_log_start":"103'490723","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1972,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 669s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7034888192,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":498202,"num_read_kb":1961760,"num_write":498395,"num_write_kb":1993580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4099,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-10-17T01:59:11.108130+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T01:59:11.108130+0000","last_peered":"2023-10-17T01:59:11.108130+0000","last_clean":"2023-10-17T01:59:11.108130+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:59:11.108130+0000","last_undegraded":"2023-10-17T01:59:11.108130+0000","last_fullsized":"2023-10-17T01:59:11.108130+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"104'488844","reported_seq":976821,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-10-17T01:59:15.083025+0000","last_change":"2023-10-17T01:59:14.644510+0000","last_active":"2023-10-17T01:59:15.083025+0000","last_peered":"2023-10-17T01:59:15.083025+0000","last_clean":"2023-10-17T01:59:15.083025+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:59:15.083025+0000","last_undegraded":"2023-10-17T01:59:15.083025+0000","last_fullsized":"2023-10-17T01:59:15.083025+0000","mapping_epoch":16,"log_start":"103'478816","ondisk_log_start":"103'478816","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6882893843,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":488864,"num_read_kb":1924456,"num_write":487571,"num_write_kb":1950278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"104'9730","reported_seq":13702,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-10-17T01:59:14.993970+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T01:59:14.993970+0000","last_peered":"2023-10-17T01:59:14.993970+0000","last_clean":"2023-10-17T01:59:14.993970+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:59:14.993970+0000","last_undegraded":"2023-10-17T01:59:14.993970+0000","last_fullsized":"2023-10-17T01:59:14.993970+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":9730,"log_dups_size":0,"ondisk_log_size":9730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":279,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-10-17T01:58:17.244435+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:58:17.244435+0000","last_peered":"2023-10-17T01:58:17.244435+0000","last_clean":"2023-10-17T01:58:17.244435+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:58:17.244435+0000","last_undegraded":"2023-10-17T01:58:17.244435+0000","last_fullsized":"2023-10-17T01:58:17.244435+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27682230291,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1963220,"num_read_kb":7730240,"num_write":1961321,"num_write_kb":7845278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16063549440,"data_stored":19464409126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40172,"ondisk_log_size":40172,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":45973,"ondisk_log_size":45973,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739297,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9665000,"kb_used_data":7841888,"kb_used_omap":670,"kb_used_meta":1822433,"kb_avail":84706840,"statfs":{"total":96636764160,"available":86739804160,"internally_reserved":0,"allocated":8030093312,"data_stored":9729921598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":686480,"internal_metadata":1866172016},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,2,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:58:17 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.36399999999999999,"15min":0.379},"min":{"1min":0.20200000000000001,"5min":0.189,"15min":0.16900000000000001},"max":{"1min":2.1600000000000001,"5min":2.5790000000000002,"15min":4.2140000000000004},"last":0.219},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.50700000000000001,"15min":0.47299999999999998},"min":{"1min":0.23300000000000001,"5min":0.19800000000000001,"15min":0.188},"max":{"1min":2.194,"5min":5.3280000000000003,"15min":5.3280000000000003},"last":0.26400000000000001}]},{"osd":1,"last update":"Tue Oct 17 01:58:42 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.54100000000000004,"15min":0.51300000000000001},"min":{"1min":0.252,"5min":0.214,"15min":0.19400000000000001},"max":{"1min":5.2169999999999996,"5min":9.5429999999999993,"15min":9.5429999999999993},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.65800000000000003,"15min":0.58599999999999997},"min":{"1min":0.27900000000000003,"5min":0.23100000000000001,"15min":0.19500000000000001},"max":{"1min":5.4379999999999997,"5min":9.5999999999999996,"15min":9.5999999999999996},"last":0.39200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739294,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5646920,"kb_used_data":3864128,"kb_used_omap":285,"kb_used_meta":1782498,"kb_avail":88724920,"statfs":{"total":96636764160,"available":90854318080,"internally_reserved":0,"allocated":3956867072,"data_stored":4794468957,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":292849,"internal_metadata":1825277967},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,6,9,4,9,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.502,"15min":0.53400000000000003},"min":{"1min":0.20599999999999999,"5min":0.16300000000000001,"15min":0.16200000000000001},"max":{"1min":0.78800000000000003,"5min":6.6890000000000001,"15min":23.177},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.55000000000000004,"15min":0.58499999999999996},"min":{"1min":0.26900000000000002,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.81299999999999994,"5min":6.7539999999999996,"15min":23.207999999999998},"last":0.36899999999999999}]},{"osd":2,"last update":"Tue Oct 17 01:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.5,"15min":0.53500000000000003},"min":{"1min":0.22600000000000001,"5min":0.192,"15min":0.184},"max":{"1min":1.0349999999999999,"5min":4.2539999999999996,"15min":13.757},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.37,"5min":0.42699999999999999,"15min":0.56000000000000005},"min":{"1min":0.24399999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.91800000000000004,"5min":4.0010000000000003,"15min":14.195},"last":0.311}]}]},{"osd":0,"up_from":8,"seq":34359739294,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5790600,"kb_used_data":3982400,"kb_used_omap":291,"kb_used_meta":1807900,"kb_avail":88581240,"statfs":{"total":96636764160,"available":90707189760,"internally_reserved":0,"allocated":4077977600,"data_stored":4940424996,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":298083,"internal_metadata":1851290525},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,2,10],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40400000000000003,"15min":0.44800000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.16200000000000001},"max":{"1min":2.02,"5min":4.1680000000000001,"15min":5.9829999999999997},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.41299999999999998,"15min":0.505},"min":{"1min":0.191,"5min":0.191,"15min":0.189},"max":{"1min":0.81599999999999995,"5min":3.1429999999999998,"15min":6.0750000000000002},"last":0.437}]},{"osd":2,"last update":"Tue Oct 17 01:58:51 2023","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.39800000000000002,"15min":0.35599999999999998},"min":{"1min":0.216,"5min":0.191,"15min":0.18099999999999999},"max":{"1min":0.52600000000000002,"5min":2.6120000000000001,"15min":2.6120000000000001},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.57299999999999995,"15min":0.57199999999999995},"min":{"1min":0.24099999999999999,"5min":0.20499999999999999,"15min":0.19400000000000001},"max":{"1min":1.026,"5min":8.827,"15min":10.378},"last":0.40799999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4077547520,"data_stored":4940144640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3956031488,"data_stored":4793864211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8029970432,"data_stored":9730400275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:59:16.858 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:59:16.858 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:59:16.859 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:59:16.859 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:59:16.859 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T01:59:31.048 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:59:31.046+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T01:59:31.067 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T01:59:31.062+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T01:59:36.861 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:59:37.326 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:59:37.327 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:59:37.370 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2427,"stamp":"2023-10-17T01:59:35.467518+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27711707992,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1993804,"num_read_kb":7777313,"num_write":1969060,"num_write_kb":7875891,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86456,"ondisk_log_size":86456,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21352948,"kb_used_data":15745244,"kb_used_omap":1247,"kb_used_meta":5606432,"kb_avail":261762572,"statfs":{"total":289910292480,"available":268044873728,"internally_reserved":0,"allocated":16123129856,"data_stored":19522994736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1277412,"internal_metadata":5740986908},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,5,4,4,10,11,11,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14741504,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3222,"num_read_kb":12864,"num_write":3629,"num_write_kb":14516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001612"},"pg_stats":[{"pgid":"2.7","version":"105'12957","reported_seq":18341,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T01:59:34.236751+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T01:59:34.236751+0000","last_peered":"2023-10-17T01:59:34.236751+0000","last_clean":"2023-10-17T01:59:34.236751+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:59:34.236751+0000","last_undegraded":"2023-10-17T01:59:34.236751+0000","last_fullsized":"2023-10-17T01:59:34.236751+0000","mapping_epoch":13,"log_start":"67'2900","ondisk_log_start":"67'2900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"105'9763","reported_seq":14974,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T01:59:34.239796+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T01:59:34.239796+0000","last_peered":"2023-10-17T01:59:34.239796+0000","last_clean":"2023-10-17T01:59:34.239796+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:59:34.239796+0000","last_undegraded":"2023-10-17T01:59:34.239796+0000","last_fullsized":"2023-10-17T01:59:34.239796+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":9763,"log_dups_size":0,"ondisk_log_size":9763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"105'6546","reported_seq":9324,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T01:59:34.211257+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T01:59:34.211257+0000","last_peered":"2023-10-17T01:59:34.211257+0000","last_clean":"2023-10-17T01:59:34.211257+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:59:34.211257+0000","last_undegraded":"2023-10-17T01:59:34.211257+0000","last_fullsized":"2023-10-17T01:59:34.211257+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6546,"log_dups_size":0,"ondisk_log_size":6546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"105'9796","reported_seq":14415,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T01:59:34.236278+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T01:59:34.236278+0000","last_peered":"2023-10-17T01:59:34.236278+0000","last_clean":"2023-10-17T01:59:34.236278+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:59:34.236278+0000","last_undegraded":"2023-10-17T01:59:34.236278+0000","last_fullsized":"2023-10-17T01:59:34.236278+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":9796,"log_dups_size":0,"ondisk_log_size":9796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"105'501207","reported_seq":999229,"reported_epoch":105,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:59:35.002113+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T01:59:35.002113+0000","last_peered":"2023-10-17T01:59:35.002113+0000","last_clean":"2023-10-17T01:59:35.002113+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:59:35.002113+0000","last_undegraded":"2023-10-17T01:59:35.002113+0000","last_fullsized":"2023-10-17T01:59:35.002113+0000","mapping_epoch":16,"log_start":"103'491110","ondisk_log_start":"103'491110","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1696,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 681s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7024500736,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":498862,"num_read_kb":1963600,"num_write":498951,"num_write_kb":1995804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":259,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-10-17T01:58:17.231082+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:58:17.231082+0000","last_peered":"2023-10-17T01:58:17.231082+0000","last_clean":"2023-10-17T01:58:17.231082+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:58:17.231082+0000","last_undegraded":"2023-10-17T01:58:17.231082+0000","last_fullsized":"2023-10-17T01:58:17.231082+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"105'482440","reported_seq":962326,"reported_epoch":105,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:59:35.387724+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T01:59:35.387724+0000","last_peered":"2023-10-17T01:59:35.387724+0000","last_clean":"2023-10-17T01:59:35.387724+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:59:35.387724+0000","last_undegraded":"2023-10-17T01:59:35.387724+0000","last_fullsized":"2023-10-17T01:59:35.387724+0000","mapping_epoch":16,"log_start":"103'472419","ondisk_log_start":"103'472419","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2184,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 673s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6754418688,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":480633,"num_read_kb":1893756,"num_write":479967,"num_write_kb":1919868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2740,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T01:59:31.587992+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T01:59:31.587992+0000","last_peered":"2023-10-17T01:59:31.587992+0000","last_clean":"2023-10-17T01:59:31.587992+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:59:31.587992+0000","last_undegraded":"2023-10-17T01:59:31.587992+0000","last_fullsized":"2023-10-17T01:59:31.587992+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"105'502602","reported_seq":1001685,"reported_epoch":105,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:59:34.973803+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T01:59:34.973803+0000","last_peered":"2023-10-17T01:59:34.973803+0000","last_clean":"2023-10-17T01:59:34.973803+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:59:34.973803+0000","last_undegraded":"2023-10-17T01:59:34.973803+0000","last_fullsized":"2023-10-17T01:59:34.973803+0000","mapping_epoch":16,"log_start":"103'492524","ondisk_log_start":"103'492524","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2028,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 689s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7042072576,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":499911,"num_read_kb":1968576,"num_write":500165,"num_write_kb":2000660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4112,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T01:59:31.585204+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T01:59:31.585204+0000","last_peered":"2023-10-17T01:59:31.585204+0000","last_clean":"2023-10-17T01:59:31.585204+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:59:31.585204+0000","last_undegraded":"2023-10-17T01:59:31.585204+0000","last_fullsized":"2023-10-17T01:59:31.585204+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"105'490620","reported_seq":980178,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T01:59:35.375834+0000","last_change":"2023-10-17T01:59:27.881799+0000","last_active":"2023-10-17T01:59:35.375834+0000","last_peered":"2023-10-17T01:59:35.375834+0000","last_clean":"2023-10-17T01:59:35.375834+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:59:35.375834+0000","last_undegraded":"2023-10-17T01:59:35.375834+0000","last_fullsized":"2023-10-17T01:59:35.375834+0000","mapping_epoch":16,"log_start":"103'480616","ondisk_log_start":"103'480616","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6890123283,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":490436,"num_read_kb":1930728,"num_write":489347,"num_write_kb":1957382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"105'9767","reported_seq":13750,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T01:59:34.283206+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T01:59:34.283206+0000","last_peered":"2023-10-17T01:59:34.283206+0000","last_clean":"2023-10-17T01:59:34.283206+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:59:34.283206+0000","last_undegraded":"2023-10-17T01:59:34.283206+0000","last_fullsized":"2023-10-17T01:59:34.283206+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":9767,"log_dups_size":0,"ondisk_log_size":9767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":280,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-10-17T01:58:39.865957+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:58:39.865957+0000","last_peered":"2023-10-17T01:58:39.865957+0000","last_clean":"2023-10-17T01:58:39.865957+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:58:39.865957+0000","last_undegraded":"2023-10-17T01:58:39.865957+0000","last_fullsized":"2023-10-17T01:58:39.865957+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27711115283,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1969842,"num_read_kb":7756660,"num_write":1968430,"num_write_kb":7873714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16121769984,"data_stored":19522629670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40200,"ondisk_log_size":40200,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":46157,"ondisk_log_size":46157,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739301,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9775700,"kb_used_data":7870348,"kb_used_omap":670,"kb_used_meta":1904673,"kb_avail":84596140,"statfs":{"total":96636764160,"available":86626447360,"internally_reserved":0,"allocated":8059236352,"data_stored":9759060409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":686480,"internal_metadata":1950385776},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,2,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.372,"15min":0.38},"min":{"1min":0.19,"5min":0.189,"15min":0.16900000000000001},"max":{"1min":0.73299999999999998,"5min":2.5790000000000002,"15min":4.2140000000000004},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.52600000000000002,"15min":0.47999999999999998},"min":{"1min":0.222,"5min":0.19800000000000001,"15min":0.188},"max":{"1min":1.5089999999999999,"5min":5.3280000000000003,"15min":5.3280000000000003},"last":0.44500000000000001}]},{"osd":1,"last update":"Tue Oct 17 01:58:42 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.54100000000000004,"15min":0.51300000000000001},"min":{"1min":0.252,"5min":0.214,"15min":0.19400000000000001},"max":{"1min":5.2169999999999996,"5min":9.5429999999999993,"15min":9.5429999999999993},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.65800000000000003,"15min":0.58599999999999997},"min":{"1min":0.27900000000000003,"5min":0.23100000000000001,"15min":0.19500000000000001},"max":{"1min":5.4379999999999997,"5min":9.5999999999999996,"15min":9.5999999999999996},"last":0.34300000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739298,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5704052,"kb_used_data":3877932,"kb_used_omap":285,"kb_used_meta":1825826,"kb_avail":88667788,"statfs":{"total":96636764160,"available":90795814912,"internally_reserved":0,"allocated":3971002368,"data_stored":4808600024,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":292849,"internal_metadata":1869645839},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,3,1,0,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.502,"15min":0.53400000000000003},"min":{"1min":0.20599999999999999,"5min":0.16300000000000001,"15min":0.16200000000000001},"max":{"1min":0.78800000000000003,"5min":6.6890000000000001,"15min":23.177},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.55000000000000004,"15min":0.58499999999999996},"min":{"1min":0.26900000000000002,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.81299999999999994,"5min":6.7539999999999996,"15min":23.207999999999998},"last":0.30299999999999999}]},{"osd":2,"last update":"Tue Oct 17 01:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.5,"15min":0.53500000000000003},"min":{"1min":0.22600000000000001,"5min":0.192,"15min":0.184},"max":{"1min":1.0349999999999999,"5min":4.2539999999999996,"15min":13.757},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.37,"5min":0.42699999999999999,"15min":0.56000000000000005},"min":{"1min":0.24399999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.91800000000000004,"5min":4.0010000000000003,"15min":14.195},"last":0.32600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739298,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5873196,"kb_used_data":3996964,"kb_used_omap":291,"kb_used_meta":1875932,"kb_avail":88498644,"statfs":{"total":96636764160,"available":90622611456,"internally_reserved":0,"allocated":4092891136,"data_stored":4955334303,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":298083,"internal_metadata":1920955293},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,7,3,9,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40400000000000003,"15min":0.44800000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.16200000000000001},"max":{"1min":2.02,"5min":4.1680000000000001,"15min":5.9829999999999997},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.41299999999999998,"15min":0.505},"min":{"1min":0.191,"5min":0.191,"15min":0.189},"max":{"1min":0.81599999999999995,"5min":3.1429999999999998,"15min":6.0750000000000002},"last":0.26300000000000001}]},{"osd":2,"last update":"Tue Oct 17 01:58:51 2023","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.39800000000000002,"15min":0.35599999999999998},"min":{"1min":0.216,"5min":0.191,"15min":0.18099999999999999},"max":{"1min":0.52600000000000002,"5min":2.6120000000000001,"15min":2.6120000000000001},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.57299999999999995,"15min":0.57199999999999995},"min":{"1min":0.24099999999999999,"5min":0.20499999999999999,"15min":0.19400000000000001},"max":{"1min":1.026,"5min":8.827,"15min":10.378},"last":0.30399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4092391424,"data_stored":4954988544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3970314240,"data_stored":4808146963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8059064320,"data_stored":9759494163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:59:37.372 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:59:37.373 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:59:37.373 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:59:37.373 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:59:37.374 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T01:59:57.375 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T01:59:57.876 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T01:59:57.877 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T01:59:57.911 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2438,"stamp":"2023-10-17T01:59:57.471069+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27740932952,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2000661,"num_read_kb":7804697,"num_write":1976249,"num_write_kb":7904647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86503,"ondisk_log_size":86503,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21588948,"kb_used_data":15802484,"kb_used_omap":1247,"kb_used_meta":5785184,"kb_avail":261526572,"statfs":{"total":289910292480,"available":267803209728,"internally_reserved":0,"allocated":16181743616,"data_stored":19581608496,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1277412,"internal_metadata":5924028956},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,6,10,15,12,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18276352,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4316,"num_read_kb":17236,"num_write":4505,"num_write_kb":18020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001946"},"pg_stats":[{"pgid":"2.7","version":"105'13001","reported_seq":18401,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T01:59:52.339183+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T01:59:52.339183+0000","last_peered":"2023-10-17T01:59:52.339183+0000","last_clean":"2023-10-17T01:59:52.339183+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T01:59:52.339183+0000","last_undegraded":"2023-10-17T01:59:52.339183+0000","last_fullsized":"2023-10-17T01:59:52.339183+0000","mapping_epoch":13,"log_start":"69'3000","ondisk_log_start":"69'3000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"105'9796","reported_seq":15019,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T01:59:52.335744+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T01:59:52.335744+0000","last_peered":"2023-10-17T01:59:52.335744+0000","last_clean":"2023-10-17T01:59:52.335744+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T01:59:52.335744+0000","last_undegraded":"2023-10-17T01:59:52.335744+0000","last_fullsized":"2023-10-17T01:59:52.335744+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":9796,"log_dups_size":0,"ondisk_log_size":9796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"105'6568","reported_seq":9355,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T01:59:52.333460+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T01:59:52.333460+0000","last_peered":"2023-10-17T01:59:52.333460+0000","last_clean":"2023-10-17T01:59:52.333460+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T01:59:52.333460+0000","last_undegraded":"2023-10-17T01:59:52.333460+0000","last_fullsized":"2023-10-17T01:59:52.333460+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6568,"log_dups_size":0,"ondisk_log_size":6568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"105'9829","reported_seq":14461,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T01:59:52.333360+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T01:59:52.333360+0000","last_peered":"2023-10-17T01:59:52.333360+0000","last_clean":"2023-10-17T01:59:52.333360+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T01:59:52.333360+0000","last_undegraded":"2023-10-17T01:59:52.333360+0000","last_fullsized":"2023-10-17T01:59:52.333360+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":9829,"log_dups_size":0,"ondisk_log_size":9829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"105'503071","reported_seq":1002808,"reported_epoch":105,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:59:55.222939+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T01:59:55.222939+0000","last_peered":"2023-10-17T01:59:55.222939+0000","last_clean":"2023-10-17T01:59:55.222939+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T01:59:55.222939+0000","last_undegraded":"2023-10-17T01:59:55.222939+0000","last_fullsized":"2023-10-17T01:59:55.222939+0000","mapping_epoch":16,"log_start":"103'493010","ondisk_log_start":"103'493010","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1752,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 702s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7031934976,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":500571,"num_read_kb":1970424,"num_write":500785,"num_write_kb":2003140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":260,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T01:59:27.284465+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:59:27.284465+0000","last_peered":"2023-10-17T01:59:27.284465+0000","last_clean":"2023-10-17T01:59:27.284465+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:59:27.284465+0000","last_undegraded":"2023-10-17T01:59:27.284465+0000","last_fullsized":"2023-10-17T01:59:27.284465+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"105'484239","reported_seq":965767,"reported_epoch":105,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:59:55.551460+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T01:59:55.551460+0000","last_peered":"2023-10-17T01:59:55.551460+0000","last_clean":"2023-10-17T01:59:55.551460+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T01:59:55.551460+0000","last_undegraded":"2023-10-17T01:59:55.551460+0000","last_fullsized":"2023-10-17T01:59:55.551460+0000","mapping_epoch":16,"log_start":"103'474219","ondisk_log_start":"103'474219","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2236,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 694s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6761582592,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":482268,"num_read_kb":1900288,"num_write":481734,"num_write_kb":1926936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2749,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T01:59:52.063881+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T01:59:52.063881+0000","last_peered":"2023-10-17T01:59:52.063881+0000","last_clean":"2023-10-17T01:59:52.063881+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T01:59:52.063881+0000","last_undegraded":"2023-10-17T01:59:52.063881+0000","last_fullsized":"2023-10-17T01:59:52.063881+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"105'504438","reported_seq":1005329,"reported_epoch":105,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T01:59:55.201599+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T01:59:55.201599+0000","last_peered":"2023-10-17T01:59:55.201599+0000","last_clean":"2023-10-17T01:59:55.201599+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T01:59:55.201599+0000","last_undegraded":"2023-10-17T01:59:55.201599+0000","last_fullsized":"2023-10-17T01:59:55.201599+0000","mapping_epoch":16,"log_start":"103'494424","ondisk_log_start":"103'494424","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2082,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 710s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7049424896,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":501703,"num_read_kb":1975728,"num_write":501970,"num_write_kb":2007880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4124,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T01:59:52.132340+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T01:59:52.132340+0000","last_peered":"2023-10-17T01:59:52.132340+0000","last_clean":"2023-10-17T01:59:52.132340+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T01:59:52.132340+0000","last_undegraded":"2023-10-17T01:59:52.132340+0000","last_fullsized":"2023-10-17T01:59:52.132340+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"105'492403","reported_seq":983683,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T01:59:55.562807+0000","last_change":"2023-10-17T01:59:49.714234+0000","last_active":"2023-10-17T01:59:55.562807+0000","last_peered":"2023-10-17T01:59:55.562807+0000","last_clean":"2023-10-17T01:59:55.562807+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T01:59:55.562807+0000","last_undegraded":"2023-10-17T01:59:55.562807+0000","last_fullsized":"2023-10-17T01:59:55.562807+0000","mapping_epoch":16,"log_start":"103'482316","ondisk_log_start":"103'482316","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6897397779,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":492157,"num_read_kb":1937604,"num_write":491130,"num_write_kb":1964514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"105'9800","reported_seq":13796,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T01:59:52.379684+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T01:59:52.379684+0000","last_peered":"2023-10-17T01:59:52.379684+0000","last_clean":"2023-10-17T01:59:52.379684+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T01:59:52.379684+0000","last_undegraded":"2023-10-17T01:59:52.379684+0000","last_fullsized":"2023-10-17T01:59:52.379684+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":9800,"log_dups_size":0,"ondisk_log_size":9800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":280,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-10-17T01:58:39.865957+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:58:39.865957+0000","last_peered":"2023-10-17T01:58:39.865957+0000","last_clean":"2023-10-17T01:58:39.865957+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:58:39.865957+0000","last_undegraded":"2023-10-17T01:58:39.865957+0000","last_fullsized":"2023-10-17T01:58:39.865957+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27740340243,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1976699,"num_read_kb":7784044,"num_write":1975619,"num_write_kb":7902470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16180477952,"data_stored":19581337638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40182,"ondisk_log_size":40182,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":46222,"ondisk_log_size":46222,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739305,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9896896,"kb_used_data":7898868,"kb_used_omap":670,"kb_used_meta":1997345,"kb_avail":84474944,"statfs":{"total":96636764160,"available":86502342656,"internally_reserved":0,"allocated":8088440832,"data_stored":9788264889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":686480,"internal_metadata":2045281904},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.372,"15min":0.38},"min":{"1min":0.19,"5min":0.189,"15min":0.16900000000000001},"max":{"1min":0.73299999999999998,"5min":2.5790000000000002,"15min":4.2140000000000004},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.52600000000000002,"15min":0.47999999999999998},"min":{"1min":0.222,"5min":0.19800000000000001,"15min":0.188},"max":{"1min":1.5089999999999999,"5min":5.3280000000000003,"15min":5.3280000000000003},"last":0.30499999999999999}]},{"osd":1,"last update":"Tue Oct 17 01:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.53600000000000003,"15min":0.49299999999999999},"min":{"1min":0.249,"5min":0.22600000000000001,"15min":0.19400000000000001},"max":{"1min":0.60699999999999998,"5min":9.5429999999999993,"15min":9.5429999999999993},"last":0.374},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.627,"15min":0.55100000000000005},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.19500000000000001},"max":{"1min":0.64100000000000001,"5min":9.5999999999999996,"15min":9.5999999999999996},"last":0.42699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739302,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5768136,"kb_used_data":3892028,"kb_used_omap":285,"kb_used_meta":1875810,"kb_avail":88603704,"statfs":{"total":96636764160,"available":90730192896,"internally_reserved":0,"allocated":3985436672,"data_stored":4823034328,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":292849,"internal_metadata":1920829455},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,3,1,11,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.502,"15min":0.53400000000000003},"min":{"1min":0.20599999999999999,"5min":0.16300000000000001,"15min":0.16200000000000001},"max":{"1min":0.78800000000000003,"5min":6.6890000000000001,"15min":23.177},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.55000000000000004,"15min":0.58499999999999996},"min":{"1min":0.26900000000000002,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.81299999999999994,"5min":6.7539999999999996,"15min":23.207999999999998},"last":0.372}]},{"osd":2,"last update":"Tue Oct 17 01:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.5,"15min":0.53500000000000003},"min":{"1min":0.22600000000000001,"5min":0.192,"15min":0.184},"max":{"1min":1.0349999999999999,"5min":4.2539999999999996,"15min":13.757},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.37,"5min":0.42699999999999999,"15min":0.56000000000000005},"min":{"1min":0.24399999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.91800000000000004,"5min":4.0010000000000003,"15min":14.195},"last":0.38400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739302,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5923916,"kb_used_data":4011588,"kb_used_omap":291,"kb_used_meta":1912028,"kb_avail":88447924,"statfs":{"total":96636764160,"available":90570674176,"internally_reserved":0,"allocated":4107866112,"data_stored":4970309279,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":298083,"internal_metadata":1957917597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,8,3,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.36799999999999999,"15min":0.45500000000000002},"min":{"1min":0.247,"5min":0.17000000000000001,"15min":0.16200000000000001},"max":{"1min":2.524,"5min":2.524,"15min":5.9829999999999997},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.38500000000000001,"15min":0.49199999999999999},"min":{"1min":0.222,"5min":0.191,"15min":0.189},"max":{"1min":0.91000000000000003,"5min":3.1429999999999998,"15min":6.0750000000000002},"last":0.219}]},{"osd":2,"last update":"Tue Oct 17 01:59:54 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.40899999999999997,"15min":0.37},"min":{"1min":0.252,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":3.008,"5min":3.008,"15min":3.008},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.54400000000000004,"15min":0.57299999999999995},"min":{"1min":0.25,"5min":0.20499999999999999,"15min":0.19400000000000001},"max":{"1min":4.5819999999999999,"5min":8.827,"15min":10.378},"last":0.25600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4107309056,"data_stored":4969906176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3984846848,"data_stored":4822679571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8088322048,"data_stored":9788751891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T01:59:57.913 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:59:57.913 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:59:57.914 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:59:57.914 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T01:59:57.914 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:00:17.915 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:00:18.472 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:00:18.474 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:00:18.516 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2448,"stamp":"2023-10-17T02:00:17.475800+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27769752408,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2007639,"num_read_kb":7832557,"num_write":1983348,"num_write_kb":7933043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86710,"ondisk_log_size":86710,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21834432,"kb_used_data":15859044,"kb_used_omap":1247,"kb_used_meta":5974112,"kb_avail":261281088,"statfs":{"total":289910292480,"available":267551834112,"internally_reserved":0,"allocated":16239661056,"data_stored":19639525936,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1277412,"internal_metadata":6117491228},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,6,7,10,12,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17817600,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4288,"num_read_kb":17132,"num_write":4385,"num_write_kb":17540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003388"},"pg_stats":[{"pgid":"2.7","version":"105'13057","reported_seq":18473,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T02:00:12.544347+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:00:12.544347+0000","last_peered":"2023-10-17T02:00:12.544347+0000","last_clean":"2023-10-17T02:00:12.544347+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:00:12.544347+0000","last_undegraded":"2023-10-17T02:00:12.544347+0000","last_fullsized":"2023-10-17T02:00:12.544347+0000","mapping_epoch":13,"log_start":"69'3000","ondisk_log_start":"69'3000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"105'9838","reported_seq":15073,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T02:00:12.543482+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:00:12.543482+0000","last_peered":"2023-10-17T02:00:12.543482+0000","last_clean":"2023-10-17T02:00:12.543482+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:00:12.543482+0000","last_undegraded":"2023-10-17T02:00:12.543482+0000","last_fullsized":"2023-10-17T02:00:12.543482+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":9838,"log_dups_size":0,"ondisk_log_size":9838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"105'6596","reported_seq":9391,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T02:00:12.543141+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:00:12.543141+0000","last_peered":"2023-10-17T02:00:12.543141+0000","last_clean":"2023-10-17T02:00:12.543141+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:00:12.543141+0000","last_undegraded":"2023-10-17T02:00:12.543141+0000","last_fullsized":"2023-10-17T02:00:12.543141+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6596,"log_dups_size":0,"ondisk_log_size":6596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"105'9871","reported_seq":14515,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T02:00:12.543324+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:00:12.543324+0000","last_peered":"2023-10-17T02:00:12.543324+0000","last_clean":"2023-10-17T02:00:12.543324+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:00:12.543324+0000","last_undegraded":"2023-10-17T02:00:12.543324+0000","last_fullsized":"2023-10-17T02:00:12.543324+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":9871,"log_dups_size":0,"ondisk_log_size":9871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"105'504943","reported_seq":1006491,"reported_epoch":105,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:00:15.446614+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T02:00:15.446614+0000","last_peered":"2023-10-17T02:00:15.446614+0000","last_clean":"2023-10-17T02:00:15.446614+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:00:15.446614+0000","last_undegraded":"2023-10-17T02:00:15.446614+0000","last_fullsized":"2023-10-17T02:00:15.446614+0000","mapping_epoch":16,"log_start":"104'494910","ondisk_log_start":"104'494910","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1808,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 722s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7039418368,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":502372,"num_read_kb":1977620,"num_write":502627,"num_write_kb":2010508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":260,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T01:59:27.284465+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:59:27.284465+0000","last_peered":"2023-10-17T01:59:27.284465+0000","last_clean":"2023-10-17T01:59:27.284465+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:59:27.284465+0000","last_undegraded":"2023-10-17T01:59:27.284465+0000","last_fullsized":"2023-10-17T01:59:27.284465+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"105'485947","reported_seq":969149,"reported_epoch":105,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:00:15.624880+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T02:00:15.624880+0000","last_peered":"2023-10-17T02:00:15.624880+0000","last_clean":"2023-10-17T02:00:15.624880+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:00:15.624880+0000","last_undegraded":"2023-10-17T02:00:15.624880+0000","last_fullsized":"2023-10-17T02:00:15.624880+0000","mapping_epoch":16,"log_start":"104'475919","ondisk_log_start":"104'475919","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2302,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 714s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6768373760,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":483936,"num_read_kb":1906948,"num_write":483405,"num_write_kb":1933620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2757,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T02:00:12.737207+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:00:12.737207+0000","last_peered":"2023-10-17T02:00:12.737207+0000","last_clean":"2023-10-17T02:00:12.737207+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:00:12.737207+0000","last_undegraded":"2023-10-17T02:00:12.737207+0000","last_fullsized":"2023-10-17T02:00:12.737207+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"105'506317","reported_seq":1009015,"reported_epoch":105,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:00:15.451327+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T02:00:15.451327+0000","last_peered":"2023-10-17T02:00:15.451327+0000","last_clean":"2023-10-17T02:00:15.451327+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:00:15.451327+0000","last_undegraded":"2023-10-17T02:00:15.451327+0000","last_fullsized":"2023-10-17T02:00:15.451327+0000","mapping_epoch":16,"log_start":"104'496224","ondisk_log_start":"104'496224","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2136,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 730s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7056932864,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":503501,"num_read_kb":1982900,"num_write":503818,"num_write_kb":2015272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4136,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T02:00:12.547588+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:00:12.547588+0000","last_peered":"2023-10-17T02:00:12.547588+0000","last_clean":"2023-10-17T02:00:12.547588+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:00:12.547588+0000","last_undegraded":"2023-10-17T02:00:12.547588+0000","last_fullsized":"2023-10-17T02:00:12.547588+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"105'494141","reported_seq":987140,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T02:00:15.626995+0000","last_change":"2023-10-17T02:00:06.700644+0000","last_active":"2023-10-17T02:00:15.626995+0000","last_peered":"2023-10-17T02:00:15.626995+0000","last_clean":"2023-10-17T02:00:15.626995+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:00:15.626995+0000","last_undegraded":"2023-10-17T02:00:15.626995+0000","last_fullsized":"2023-10-17T02:00:15.626995+0000","mapping_epoch":16,"log_start":"104'484116","ondisk_log_start":"104'484116","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6904434707,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":493868,"num_read_kb":1944436,"num_write":492868,"num_write_kb":1971466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"105'9842","reported_seq":13850,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T02:00:12.543743+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:00:12.543743+0000","last_peered":"2023-10-17T02:00:12.543743+0000","last_clean":"2023-10-17T02:00:12.543743+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:00:12.543743+0000","last_undegraded":"2023-10-17T02:00:12.543743+0000","last_fullsized":"2023-10-17T02:00:12.543743+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":9842,"log_dups_size":0,"ondisk_log_size":9842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":280,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-10-17T01:58:39.865957+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:58:39.865957+0000","last_peered":"2023-10-17T01:58:39.865957+0000","last_clean":"2023-10-17T01:58:39.865957+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:58:39.865957+0000","last_undegraded":"2023-10-17T01:58:39.865957+0000","last_fullsized":"2023-10-17T01:58:39.865957+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27769159699,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1983677,"num_read_kb":7811904,"num_write":1982718,"num_write_kb":7930866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16238542848,"data_stored":19639402534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40179,"ondisk_log_size":40179,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":46432,"ondisk_log_size":46432,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739309,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10034940,"kb_used_data":7927280,"kb_used_omap":670,"kb_used_meta":2106977,"kb_avail":84336900,"statfs":{"total":96636764160,"available":86360985600,"internally_reserved":0,"allocated":8117534720,"data_stored":9817358777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":686480,"internal_metadata":2157545072},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 01:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.372,"15min":0.38},"min":{"1min":0.19,"5min":0.189,"15min":0.16900000000000001},"max":{"1min":0.73299999999999998,"5min":2.5790000000000002,"15min":4.2140000000000004},"last":0.246},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.52600000000000002,"15min":0.47999999999999998},"min":{"1min":0.222,"5min":0.19800000000000001,"15min":0.188},"max":{"1min":1.5089999999999999,"5min":5.3280000000000003,"15min":5.3280000000000003},"last":0.31900000000000001}]},{"osd":1,"last update":"Tue Oct 17 01:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.53600000000000003,"15min":0.49299999999999999},"min":{"1min":0.249,"5min":0.22600000000000001,"15min":0.19400000000000001},"max":{"1min":0.60699999999999998,"5min":9.5429999999999993,"15min":9.5429999999999993},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.627,"15min":0.55100000000000005},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.19500000000000001},"max":{"1min":0.64100000000000001,"5min":9.5999999999999996,"15min":9.5999999999999996},"last":0.26200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739306,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5799720,"kb_used_data":3905568,"kb_used_omap":285,"kb_used_meta":1893858,"kb_avail":88572120,"statfs":{"total":96636764160,"available":90697850880,"internally_reserved":0,"allocated":3999301632,"data_stored":4836899288,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":292849,"internal_metadata":1939310607},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:00:09 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.40999999999999998,"15min":0.54300000000000004},"min":{"1min":0.18099999999999999,"5min":0.16300000000000001,"15min":0.16200000000000001},"max":{"1min":2.4809999999999999,"5min":4.141,"15min":23.177},"last":0.501},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45900000000000002,"15min":0.59399999999999997},"min":{"1min":0.248,"5min":0.22,"15min":0.216},"max":{"1min":2.512,"5min":4.1820000000000004,"15min":23.207999999999998},"last":0.53800000000000003}]},{"osd":2,"last update":"Tue Oct 17 02:00:09 2023","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.51100000000000001,"15min":0.46899999999999997},"min":{"1min":0.216,"5min":0.19700000000000001,"15min":0.184},"max":{"1min":1.893,"5min":4.2539999999999996,"15min":5.423},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.432,"15min":0.47299999999999998},"min":{"1min":0.25900000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.9279999999999999,"5min":4.0010000000000003,"15min":5.4569999999999999},"last":0.317}]}]},{"osd":0,"up_from":8,"seq":34359739306,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5999772,"kb_used_data":4026196,"kb_used_omap":291,"kb_used_meta":1973276,"kb_avail":88372068,"statfs":{"total":96636764160,"available":90492997632,"internally_reserved":0,"allocated":4122824704,"data_stored":4985267871,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":298083,"internal_metadata":2020635549},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,4,6,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.36799999999999999,"15min":0.45500000000000002},"min":{"1min":0.247,"5min":0.17000000000000001,"15min":0.16200000000000001},"max":{"1min":2.524,"5min":2.524,"15min":5.9829999999999997},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.38500000000000001,"15min":0.49199999999999999},"min":{"1min":0.222,"5min":0.191,"15min":0.189},"max":{"1min":0.91000000000000003,"5min":3.1429999999999998,"15min":6.0750000000000002},"last":0.34300000000000003}]},{"osd":2,"last update":"Tue Oct 17 01:59:54 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.40899999999999997,"15min":0.37},"min":{"1min":0.252,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":3.008,"5min":3.008,"15min":3.008},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.54400000000000004,"15min":0.57299999999999995},"min":{"1min":0.25,"5min":0.20499999999999999,"15min":0.19400000000000001},"max":{"1min":4.5819999999999999,"5min":8.827,"15min":10.378},"last":0.36899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4122402816,"data_stored":4984999936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3998810112,"data_stored":4836642835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8117329920,"data_stored":9817759763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:00:18.518 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:00:18.519 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:00:18.519 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:00:18.520 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:00:18.520 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:00:38.521 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:00:39.081 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:00:39.082 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:00:39.104 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2459,"stamp":"2023-10-17T02:00:37.479006+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27799071576,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2014452,"num_read_kb":7859745,"num_write":1990527,"num_write_kb":7961759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86810,"ondisk_log_size":86810,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21571080,"kb_used_data":15917552,"kb_used_omap":1372,"kb_used_meta":5652131,"kb_avail":261544440,"statfs":{"total":289910292480,"available":267821506560,"internally_reserved":0,"allocated":16299573248,"data_stored":19699438128,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1405415,"internal_metadata":5787782681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,12,7,19,18],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14741504,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3499,"num_read_kb":13968,"num_write":3590,"num_write_kb":14360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001612"},"pg_stats":[{"pgid":"2.7","version":"105'13097","reported_seq":18529,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T02:00:35.343081+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:00:35.343081+0000","last_peered":"2023-10-17T02:00:35.343081+0000","last_clean":"2023-10-17T02:00:35.343081+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:00:35.343081+0000","last_undegraded":"2023-10-17T02:00:35.343081+0000","last_fullsized":"2023-10-17T02:00:35.343081+0000","mapping_epoch":13,"log_start":"69'3000","ondisk_log_start":"69'3000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"105'9868","reported_seq":15116,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T02:00:35.332585+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:00:35.332585+0000","last_peered":"2023-10-17T02:00:35.332585+0000","last_clean":"2023-10-17T02:00:35.332585+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:00:35.332585+0000","last_undegraded":"2023-10-17T02:00:35.332585+0000","last_fullsized":"2023-10-17T02:00:35.332585+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":9868,"log_dups_size":0,"ondisk_log_size":9868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"105'6616","reported_seq":9419,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T02:00:35.326019+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:00:35.326019+0000","last_peered":"2023-10-17T02:00:35.326019+0000","last_clean":"2023-10-17T02:00:35.326019+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:00:35.326019+0000","last_undegraded":"2023-10-17T02:00:35.326019+0000","last_fullsized":"2023-10-17T02:00:35.326019+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6616,"log_dups_size":0,"ondisk_log_size":6616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"105'9901","reported_seq":14557,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T02:00:35.338267+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:00:35.338267+0000","last_peered":"2023-10-17T02:00:35.338267+0000","last_clean":"2023-10-17T02:00:35.338267+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:00:35.338267+0000","last_undegraded":"2023-10-17T02:00:35.338267+0000","last_fullsized":"2023-10-17T02:00:35.338267+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":9901,"log_dups_size":0,"ondisk_log_size":9901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"105'506823","reported_seq":1010068,"reported_epoch":105,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:00:35.612563+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T02:00:35.612563+0000","last_peered":"2023-10-17T02:00:35.612563+0000","last_clean":"2023-10-17T02:00:35.612563+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:00:35.612563+0000","last_undegraded":"2023-10-17T02:00:35.612563+0000","last_fullsized":"2023-10-17T02:00:35.612563+0000","mapping_epoch":16,"log_start":"104'496810","ondisk_log_start":"104'496810","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1846,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 742s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7047061504,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":504064,"num_read_kb":1984376,"num_write":504487,"num_write_kb":2017948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":260,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T01:59:27.284465+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:59:27.284465+0000","last_peered":"2023-10-17T01:59:27.284465+0000","last_clean":"2023-10-17T01:59:27.284465+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:59:27.284465+0000","last_undegraded":"2023-10-17T01:59:27.284465+0000","last_fullsized":"2023-10-17T01:59:27.284465+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"105'487749","reported_seq":972658,"reported_epoch":105,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:00:35.740947+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T02:00:35.740947+0000","last_peered":"2023-10-17T02:00:35.740947+0000","last_clean":"2023-10-17T02:00:35.740947+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:00:35.740947+0000","last_undegraded":"2023-10-17T02:00:35.740947+0000","last_fullsized":"2023-10-17T02:00:35.740947+0000","mapping_epoch":16,"log_start":"104'477719","ondisk_log_start":"104'477719","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2336,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 734s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6775635968,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":485630,"num_read_kb":1913712,"num_write":485188,"num_write_kb":1940752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2763,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T02:00:27.920363+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:00:27.920363+0000","last_peered":"2023-10-17T02:00:27.920363+0000","last_clean":"2023-10-17T02:00:27.920363+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:00:27.920363+0000","last_undegraded":"2023-10-17T02:00:27.920363+0000","last_fullsized":"2023-10-17T02:00:27.920363+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"105'508149","reported_seq":1012569,"reported_epoch":105,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:00:35.587985+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T02:00:35.587985+0000","last_peered":"2023-10-17T02:00:35.587985+0000","last_clean":"2023-10-17T02:00:35.587985+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:00:35.587985+0000","last_undegraded":"2023-10-17T02:00:35.587985+0000","last_fullsized":"2023-10-17T02:00:35.587985+0000","mapping_epoch":16,"log_start":"104'498124","ondisk_log_start":"104'498124","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2196,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 750s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7064268800,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":505216,"num_read_kb":1989736,"num_write":505618,"num_write_kb":2022472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4148,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T02:00:33.054729+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:00:33.054729+0000","last_peered":"2023-10-17T02:00:33.054729+0000","last_clean":"2023-10-17T02:00:33.054729+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:00:33.054729+0000","last_undegraded":"2023-10-17T02:00:33.054729+0000","last_fullsized":"2023-10-17T02:00:33.054729+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"105'495877","reported_seq":990594,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T02:00:35.745879+0000","last_change":"2023-10-17T02:00:25.708459+0000","last_active":"2023-10-17T02:00:35.745879+0000","last_peered":"2023-10-17T02:00:35.745879+0000","last_clean":"2023-10-17T02:00:35.745879+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:00:35.745879+0000","last_undegraded":"2023-10-17T02:00:35.745879+0000","last_fullsized":"2023-10-17T02:00:35.745879+0000","mapping_epoch":16,"log_start":"104'485816","ondisk_log_start":"104'485816","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6911512595,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":495580,"num_read_kb":1951268,"num_write":494604,"num_write_kb":1978410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"105'9872","reported_seq":13892,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T02:00:35.374500+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:00:35.374500+0000","last_peered":"2023-10-17T02:00:35.374500+0000","last_clean":"2023-10-17T02:00:35.374500+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:00:35.374500+0000","last_undegraded":"2023-10-17T02:00:35.374500+0000","last_fullsized":"2023-10-17T02:00:35.374500+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":9872,"log_dups_size":0,"ondisk_log_size":9872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":280,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-10-17T01:58:39.865957+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T01:58:39.865957+0000","last_peered":"2023-10-17T01:58:39.865957+0000","last_clean":"2023-10-17T01:58:39.865957+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T01:58:39.865957+0000","last_undegraded":"2023-10-17T01:58:39.865957+0000","last_fullsized":"2023-10-17T01:58:39.865957+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27798478867,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1990490,"num_read_kb":7839092,"num_write":1989897,"num_write_kb":7959582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16297422848,"data_stored":19698282534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40129,"ondisk_log_size":40129,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":46582,"ondisk_log_size":46582,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739313,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10131480,"kb_used_data":7955792,"kb_used_omap":670,"kb_used_meta":2175009,"kb_avail":84240360,"statfs":{"total":96636764160,"available":86262128640,"internally_reserved":0,"allocated":8146731008,"data_stored":9846555065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":686480,"internal_metadata":2227209840},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,5,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:00:19 2023","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.35099999999999998,"15min":0.379},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.16900000000000001},"max":{"1min":0.56599999999999995,"5min":2.1600000000000001,"15min":4.2140000000000004},"last":0.316},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.505,"15min":0.48199999999999998},"min":{"1min":0.28000000000000003,"5min":0.218,"15min":0.188},"max":{"1min":0.78200000000000003,"5min":5.3280000000000003,"15min":5.3280000000000003},"last":0.38900000000000001}]},{"osd":1,"last update":"Tue Oct 17 01:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.53600000000000003,"15min":0.49299999999999999},"min":{"1min":0.249,"5min":0.22600000000000001,"15min":0.19400000000000001},"max":{"1min":0.60699999999999998,"5min":9.5429999999999993,"15min":9.5429999999999993},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.627,"15min":0.55100000000000005},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.19500000000000001},"max":{"1min":0.64100000000000001,"5min":9.5999999999999996,"15min":9.5999999999999996},"last":0.28000000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739310,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5886700,"kb_used_data":3920356,"kb_used_omap":285,"kb_used_meta":1966050,"kb_avail":88485140,"statfs":{"total":96636764160,"available":90608783360,"internally_reserved":0,"allocated":4014444544,"data_stored":4852042200,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":292849,"internal_metadata":2013235215},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1,6,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:00:09 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.40999999999999998,"15min":0.54300000000000004},"min":{"1min":0.18099999999999999,"5min":0.16300000000000001,"15min":0.16200000000000001},"max":{"1min":2.4809999999999999,"5min":4.141,"15min":23.177},"last":0.254},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45900000000000002,"15min":0.59399999999999997},"min":{"1min":0.248,"5min":0.22,"15min":0.216},"max":{"1min":2.512,"5min":4.1820000000000004,"15min":23.207999999999998},"last":0.375}]},{"osd":2,"last update":"Tue Oct 17 02:00:09 2023","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.51100000000000001,"15min":0.46899999999999997},"min":{"1min":0.216,"5min":0.19700000000000001,"15min":0.184},"max":{"1min":1.893,"5min":4.2539999999999996,"15min":5.423},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.432,"15min":0.47299999999999998},"min":{"1min":0.25900000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.9279999999999999,"5min":4.0010000000000003,"15min":5.4569999999999999},"last":0.33000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739310,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5552900,"kb_used_data":4041404,"kb_used_omap":416,"kb_used_meta":1511071,"kb_avail":88818940,"statfs":{"total":96636764160,"available":90950594560,"internally_reserved":0,"allocated":4138397696,"data_stored":5000840863,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":426086,"internal_metadata":1547337626},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,8,4,8,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 01:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.36799999999999999,"15min":0.45500000000000002},"min":{"1min":0.247,"5min":0.17000000000000001,"15min":0.16200000000000001},"max":{"1min":2.524,"5min":2.524,"15min":5.9829999999999997},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.38500000000000001,"15min":0.49199999999999999},"min":{"1min":0.222,"5min":0.191,"15min":0.189},"max":{"1min":0.91000000000000003,"5min":3.1429999999999998,"15min":6.0750000000000002},"last":0.46899999999999997}]},{"osd":2,"last update":"Tue Oct 17 01:59:54 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.40899999999999997,"15min":0.37},"min":{"1min":0.252,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":3.008,"5min":3.008,"15min":3.008},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.54400000000000004,"15min":0.57299999999999995},"min":{"1min":0.25,"5min":0.20499999999999999,"15min":0.19400000000000001},"max":{"1min":4.5819999999999999,"5min":8.827,"15min":10.378},"last":0.50800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4137402368,"data_stored":4999999488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4013228032,"data_stored":4851060755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8146792448,"data_stored":9847222291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:00:39.106 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:00:39.106 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:00:39.107 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:00:39.107 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:00:39.108 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:00:53.594 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:00:53.588+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T02:00:53.604 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:00:53.600+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T02:00:53.611 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:00:53.608+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T02:00:53.614 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:00:53.608+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T02:00:53.615 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:00:53.612+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T02:00:53.615 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:00:53.612+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T02:00:53.615 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:00:53.612+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T02:00:53.623 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:00:53.620+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T02:00:53.624 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:00:53.620+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T02:00:53.624 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:00:53.620+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T02:00:53.624 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:00:53.620+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T02:00:53.625 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:00:53.620+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T02:00:53.625 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:00:53.620+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T02:00:53.625 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:00:53.620+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T02:00:59.108 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:00:59.608 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:00:59.609 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:00:59.640 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2470,"stamp":"2023-10-17T02:00:59.483056+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27828038488,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2021280,"num_read_kb":7887017,"num_write":1997609,"num_write_kb":7990087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87020,"ondisk_log_size":87020,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20742040,"kb_used_data":15981056,"kb_used_omap":1466,"kb_used_meta":4759493,"kb_avail":262373480,"statfs":{"total":289910292480,"available":268670443520,"internally_reserved":0,"allocated":16364601344,"data_stored":19764453408,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1501574,"internal_metadata":4873721466},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,1,4,6,9,20,11,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14409728,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3553,"num_read_kb":14180,"num_write":3513,"num_write_kb":14052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002409"},"pg_stats":[{"pgid":"2.7","version":"106'13137","reported_seq":18587,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-17T02:00:54.292673+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:00:54.292673+0000","last_peered":"2023-10-17T02:00:54.292673+0000","last_clean":"2023-10-17T02:00:54.292673+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:00:54.292673+0000","last_undegraded":"2023-10-17T02:00:54.292673+0000","last_fullsized":"2023-10-17T02:00:54.292673+0000","mapping_epoch":13,"log_start":"69'3100","ondisk_log_start":"69'3100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"106'9898","reported_seq":15159,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-17T02:00:54.292451+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:00:54.292451+0000","last_peered":"2023-10-17T02:00:54.292451+0000","last_clean":"2023-10-17T02:00:54.292451+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:00:54.292451+0000","last_undegraded":"2023-10-17T02:00:54.292451+0000","last_fullsized":"2023-10-17T02:00:54.292451+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":9898,"log_dups_size":0,"ondisk_log_size":9898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"106'6636","reported_seq":9448,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-17T02:00:54.269833+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:00:54.269833+0000","last_peered":"2023-10-17T02:00:54.269833+0000","last_clean":"2023-10-17T02:00:54.269833+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:00:54.269833+0000","last_undegraded":"2023-10-17T02:00:54.269833+0000","last_fullsized":"2023-10-17T02:00:54.269833+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6636,"log_dups_size":0,"ondisk_log_size":6636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"106'9931","reported_seq":14600,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-17T02:00:54.282761+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:00:54.282761+0000","last_peered":"2023-10-17T02:00:54.282761+0000","last_clean":"2023-10-17T02:00:54.282761+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:00:54.282761+0000","last_undegraded":"2023-10-17T02:00:54.282761+0000","last_fullsized":"2023-10-17T02:00:54.282761+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":9931,"log_dups_size":0,"ondisk_log_size":9931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"106'508591","reported_seq":1013561,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:00:55.740470+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T02:00:55.740470+0000","last_peered":"2023-10-17T02:00:55.740470+0000","last_clean":"2023-10-17T02:00:55.740470+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:00:55.740470+0000","last_undegraded":"2023-10-17T02:00:55.740470+0000","last_fullsized":"2023-10-17T02:00:55.740470+0000","mapping_epoch":16,"log_start":"104'498510","ondisk_log_start":"104'498510","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1896,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 762s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7054200832,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":505774,"num_read_kb":1991212,"num_write":506229,"num_write_kb":2024916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":261,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T01:59:45.085795+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:59:45.085795+0000","last_peered":"2023-10-17T01:59:45.085795+0000","last_clean":"2023-10-17T01:59:45.085795+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:59:45.085795+0000","last_undegraded":"2023-10-17T01:59:45.085795+0000","last_fullsized":"2023-10-17T01:59:45.085795+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"106'489577","reported_seq":976174,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:00:55.891916+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T02:00:55.891916+0000","last_peered":"2023-10-17T02:00:55.891916+0000","last_clean":"2023-10-17T02:00:55.891916+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:00:55.891916+0000","last_undegraded":"2023-10-17T02:00:55.891916+0000","last_fullsized":"2023-10-17T02:00:55.891916+0000","mapping_epoch":16,"log_start":"104'479519","ondisk_log_start":"104'479519","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2398,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 754s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6782922752,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":487311,"num_read_kb":1920420,"num_write":486981,"num_write_kb":1947924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2774,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-17T02:00:53.556780+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:00:53.556780+0000","last_peered":"2023-10-17T02:00:53.556780+0000","last_clean":"2023-10-17T02:00:53.556780+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:00:53.556780+0000","last_undegraded":"2023-10-17T02:00:53.556780+0000","last_fullsized":"2023-10-17T02:00:53.556780+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"106'509978","reported_seq":1016114,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:00:55.717333+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T02:00:55.717333+0000","last_peered":"2023-10-17T02:00:55.717333+0000","last_clean":"2023-10-17T02:00:55.717333+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:00:55.717333+0000","last_undegraded":"2023-10-17T02:00:55.717333+0000","last_fullsized":"2023-10-17T02:00:55.717333+0000","mapping_epoch":16,"log_start":"104'499925","ondisk_log_start":"104'499925","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2230,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 770s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7071645696,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":506921,"num_read_kb":1996544,"num_write":507429,"num_write_kb":2029716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4162,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-17T02:00:53.541570+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:00:53.541570+0000","last_peered":"2023-10-17T02:00:53.541570+0000","last_clean":"2023-10-17T02:00:53.541570+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:00:53.541570+0000","last_undegraded":"2023-10-17T02:00:53.541570+0000","last_fullsized":"2023-10-17T02:00:53.541570+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"106'497613","reported_seq":994071,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-17T02:00:55.887004+0000","last_change":"2023-10-17T02:00:54.857964+0000","last_active":"2023-10-17T02:00:55.887004+0000","last_peered":"2023-10-17T02:00:55.887004+0000","last_clean":"2023-10-17T02:00:55.887004+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:00:55.887004+0000","last_undegraded":"2023-10-17T02:00:55.887004+0000","last_fullsized":"2023-10-17T02:00:55.887004+0000","mapping_epoch":16,"log_start":"104'487516","ondisk_log_start":"104'487516","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6918676499,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":497312,"num_read_kb":1958188,"num_write":496340,"num_write_kb":1985354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"106'9902","reported_seq":13935,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-17T02:00:54.330352+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:00:54.330352+0000","last_peered":"2023-10-17T02:00:54.330352+0000","last_clean":"2023-10-17T02:00:54.330352+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:00:54.330352+0000","last_undegraded":"2023-10-17T02:00:54.330352+0000","last_fullsized":"2023-10-17T02:00:54.330352+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":9902,"log_dups_size":0,"ondisk_log_size":9902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":282,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-17T02:00:37.357112+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:00:37.357112+0000","last_peered":"2023-10-17T02:00:37.357112+0000","last_clean":"2023-10-17T02:00:37.357112+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:00:37.357112+0000","last_undegraded":"2023-10-17T02:00:37.357112+0000","last_fullsized":"2023-10-17T02:00:37.357112+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27827445779,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":1997318,"num_read_kb":7866364,"num_write":1996979,"num_write_kb":7987910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16362586112,"data_stored":19763445798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40289,"ondisk_log_size":40289,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":46632,"ondisk_log_size":46632,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739318,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9210356,"kb_used_data":7991084,"kb_used_omap":764,"kb_used_meta":1218499,"kb_avail":85161484,"statfs":{"total":96636764160,"available":87205359616,"internally_reserved":0,"allocated":8182870016,"data_stored":9882689801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":782639,"internal_metadata":1247743697},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,2,5,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:00:19 2023","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.35099999999999998,"15min":0.379},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.16900000000000001},"max":{"1min":0.56599999999999995,"5min":2.1600000000000001,"15min":4.2140000000000004},"last":0.192},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.505,"15min":0.48199999999999998},"min":{"1min":0.28000000000000003,"5min":0.218,"15min":0.188},"max":{"1min":0.78200000000000003,"5min":5.3280000000000003,"15min":5.3280000000000003},"last":0.27300000000000002}]},{"osd":1,"last update":"Tue Oct 17 02:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.52900000000000003,"15min":0.49099999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.19400000000000001},"max":{"1min":0.46899999999999997,"5min":9.5429999999999993,"15min":9.5429999999999993},"last":0.318},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.57099999999999995,"15min":0.55000000000000004},"min":{"1min":0.26200000000000001,"5min":0.23100000000000001,"15min":0.19500000000000001},"max":{"1min":0.503,"5min":9.5999999999999996,"15min":9.5999999999999996},"last":0.34799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739314,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5932752,"kb_used_data":3934472,"kb_used_omap":285,"kb_used_meta":1997986,"kb_avail":88439088,"statfs":{"total":96636764160,"available":90561626112,"internally_reserved":0,"allocated":4028899328,"data_stored":4866492712,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":292849,"internal_metadata":2045937679},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,1,4,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:00:09 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.40999999999999998,"15min":0.54300000000000004},"min":{"1min":0.18099999999999999,"5min":0.16300000000000001,"15min":0.16200000000000001},"max":{"1min":2.4809999999999999,"5min":4.141,"15min":23.177},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45900000000000002,"15min":0.59399999999999997},"min":{"1min":0.248,"5min":0.22,"15min":0.216},"max":{"1min":2.512,"5min":4.1820000000000004,"15min":23.207999999999998},"last":0.29999999999999999}]},{"osd":2,"last update":"Tue Oct 17 02:00:09 2023","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.51100000000000001,"15min":0.46899999999999997},"min":{"1min":0.216,"5min":0.19700000000000001,"15min":0.184},"max":{"1min":1.893,"5min":4.2539999999999996,"15min":5.423},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.432,"15min":0.47299999999999998},"min":{"1min":0.25900000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.9279999999999999,"5min":4.0010000000000003,"15min":5.4569999999999999},"last":0.255}]}]},{"osd":0,"up_from":8,"seq":34359739314,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5598932,"kb_used_data":4055500,"kb_used_omap":416,"kb_used_meta":1543007,"kb_avail":88772908,"statfs":{"total":96636764160,"available":90903457792,"internally_reserved":0,"allocated":4152832000,"data_stored":5015270895,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":426086,"internal_metadata":1580040090},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,3,7,11,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.436,"15min":0.44500000000000001},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.16200000000000001},"max":{"1min":4.101,"5min":4.101,"15min":5.9829999999999997},"last":0.313},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.40799999999999997,"15min":0.48599999999999999},"min":{"1min":0.20499999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.2970000000000002,"5min":3.1429999999999998,"15min":6.0750000000000002},"last":0.224}]},{"osd":2,"last update":"Tue Oct 17 01:59:54 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.40899999999999997,"15min":0.37},"min":{"1min":0.252,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":3.008,"5min":3.008,"15min":3.008},"last":0.249},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.54400000000000004,"15min":0.57299999999999995},"min":{"1min":0.25,"5min":0.20499999999999999,"15min":0.19400000000000001},"max":{"1min":4.5819999999999999,"5min":8.827,"15min":10.378},"last":0.35099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4151959552,"data_stored":5014556672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4027691008,"data_stored":4865523731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8182935552,"data_stored":9883365395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:00:59.642 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:00:59.642 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:00:59.642 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:00:59.643 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:00:59.643 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-17T02:01:00.199 INFO:teuthology.orchestra.run.smithi169.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-17T02:01:00.256 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-17T02:01:00.829 INFO:teuthology.orchestra.run.smithi169.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-17T02:01:00.888 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-17T02:01:01.458 INFO:teuthology.orchestra.run.smithi169.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-17T02:01:01.494 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-17T02:01:02.031 INFO:teuthology.orchestra.run.smithi169.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-17T02:01:02.084 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:01:22.085 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:01:22.661 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:01:22.663 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:01:22.700 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2481,"stamp":"2023-10-17T02:01:21.486748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27863415640,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2030607,"num_read_kb":7924217,"num_write":2006230,"num_write_kb":8024571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87309,"ondisk_log_size":87309,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20310860,"kb_used_data":16042292,"kb_used_omap":1542,"kb_used_meta":4267001,"kb_avail":262804660,"statfs":{"total":289910292480,"available":269111971840,"internally_reserved":0,"allocated":16427307008,"data_stored":19827159072,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1579788,"internal_metadata":4369409268},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,2,13,16,23,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21655552,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5649,"num_read_kb":22532,"num_write":5271,"num_write_kb":21084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001888"},"pg_stats":[{"pgid":"2.7","version":"106'13197","reported_seq":18667,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-17T02:01:20.840337+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:01:20.840337+0000","last_peered":"2023-10-17T02:01:20.840337+0000","last_clean":"2023-10-17T02:01:20.840337+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:01:20.840337+0000","last_undegraded":"2023-10-17T02:01:20.840337+0000","last_fullsized":"2023-10-17T02:01:20.840337+0000","mapping_epoch":13,"log_start":"69'3100","ondisk_log_start":"69'3100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"106'9943","reported_seq":15219,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-17T02:01:20.836799+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:01:20.836799+0000","last_peered":"2023-10-17T02:01:20.836799+0000","last_clean":"2023-10-17T02:01:20.836799+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:01:20.836799+0000","last_undegraded":"2023-10-17T02:01:20.836799+0000","last_fullsized":"2023-10-17T02:01:20.836799+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":9943,"log_dups_size":0,"ondisk_log_size":9943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"106'6666","reported_seq":9488,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-17T02:01:20.820531+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:01:20.820531+0000","last_peered":"2023-10-17T02:01:20.820531+0000","last_clean":"2023-10-17T02:01:20.820531+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:01:20.820531+0000","last_undegraded":"2023-10-17T02:01:20.820531+0000","last_fullsized":"2023-10-17T02:01:20.820531+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6666,"log_dups_size":0,"ondisk_log_size":6666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"106'9976","reported_seq":14660,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-17T02:01:20.820643+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:01:20.820643+0000","last_peered":"2023-10-17T02:01:20.820643+0000","last_clean":"2023-10-17T02:01:20.820643+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:01:20.820643+0000","last_undegraded":"2023-10-17T02:01:20.820643+0000","last_fullsized":"2023-10-17T02:01:20.820643+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":9976,"log_dups_size":0,"ondisk_log_size":9976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"106'510896","reported_seq":1018268,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:01:20.917114+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T02:01:20.917114+0000","last_peered":"2023-10-17T02:01:20.917114+0000","last_clean":"2023-10-17T02:01:20.917114+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:01:20.917114+0000","last_undegraded":"2023-10-17T02:01:20.917114+0000","last_fullsized":"2023-10-17T02:01:20.917114+0000","mapping_epoch":16,"log_start":"105'500810","ondisk_log_start":"105'500810","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1994,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 787s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7063425024,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":508165,"num_read_kb":2000764,"num_write":508477,"num_write_kb":2033908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":261,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T01:59:45.085795+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:59:45.085795+0000","last_peered":"2023-10-17T01:59:45.085795+0000","last_clean":"2023-10-17T01:59:45.085795+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:59:45.085795+0000","last_undegraded":"2023-10-17T01:59:45.085795+0000","last_fullsized":"2023-10-17T01:59:45.085795+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"106'491693","reported_seq":980565,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:01:21.054714+0000","last_change":"2023-10-17T01:48:21.494303+0000","last_active":"2023-10-17T02:01:21.054714+0000","last_peered":"2023-10-17T02:01:21.054714+0000","last_clean":"2023-10-17T02:01:21.054714+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:01:21.054714+0000","last_undegraded":"2023-10-17T02:01:21.054714+0000","last_fullsized":"2023-10-17T02:01:21.054714+0000","mapping_epoch":16,"log_start":"104'481619","ondisk_log_start":"104'481619","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2472,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 779s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6791446528,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":489571,"num_read_kb":1929432,"num_write":489056,"num_write_kb":1956224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2782,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-17T02:01:13.988128+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:01:13.988128+0000","last_peered":"2023-10-17T02:01:13.988128+0000","last_clean":"2023-10-17T02:01:13.988128+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:01:13.988128+0000","last_undegraded":"2023-10-17T02:01:13.988128+0000","last_fullsized":"2023-10-17T02:01:13.988128+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"106'512225","reported_seq":1020803,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:01:20.925142+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T02:01:20.925142+0000","last_peered":"2023-10-17T02:01:20.925142+0000","last_clean":"2023-10-17T02:01:20.925142+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:01:20.925142+0000","last_undegraded":"2023-10-17T02:01:20.925142+0000","last_fullsized":"2023-10-17T02:01:20.925142+0000","mapping_epoch":16,"log_start":"105'502125","ondisk_log_start":"105'502125","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2314,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 795s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7080677376,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":509346,"num_read_kb":2006204,"num_write":509631,"num_write_kb":2038524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4174,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-17T02:01:14.013137+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:01:14.013137+0000","last_peered":"2023-10-17T02:01:14.013137+0000","last_clean":"2023-10-17T02:01:14.013137+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:01:14.013137+0000","last_undegraded":"2023-10-17T02:01:14.013137+0000","last_fullsized":"2023-10-17T02:01:14.013137+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"106'499709","reported_seq":998421,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-17T02:01:21.065774+0000","last_change":"2023-10-17T02:01:19.327978+0000","last_active":"2023-10-17T02:01:21.065774+0000","last_peered":"2023-10-17T02:01:21.065774+0000","last_clean":"2023-10-17T02:01:21.065774+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:01:21.065774+0000","last_undegraded":"2023-10-17T02:01:21.065774+0000","last_fullsized":"2023-10-17T02:01:21.065774+0000","mapping_epoch":16,"log_start":"104'489616","ondisk_log_start":"104'489616","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6927274003,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":499563,"num_read_kb":1967164,"num_write":498436,"num_write_kb":1993738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"106'9947","reported_seq":13995,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-17T02:01:20.873795+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:01:20.873795+0000","last_peered":"2023-10-17T02:01:20.873795+0000","last_clean":"2023-10-17T02:01:20.873795+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:01:20.873795+0000","last_undegraded":"2023-10-17T02:01:20.873795+0000","last_fullsized":"2023-10-17T02:01:20.873795+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":9947,"log_dups_size":0,"ondisk_log_size":9947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":282,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-17T02:00:37.357112+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:00:37.357112+0000","last_peered":"2023-10-17T02:00:37.357112+0000","last_clean":"2023-10-17T02:00:37.357112+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:00:37.357112+0000","last_undegraded":"2023-10-17T02:00:37.357112+0000","last_fullsized":"2023-10-17T02:00:37.357112+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27862822931,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2006645,"num_read_kb":7903564,"num_write":2005600,"num_write_kb":8022394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16425951232,"data_stored":19826810918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40353,"ondisk_log_size":40353,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":46857,"ondisk_log_size":46857,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739322,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9318040,"kb_used_data":8018256,"kb_used_omap":764,"kb_used_meta":1299011,"kb_avail":85053800,"statfs":{"total":96636764160,"available":87095091200,"internally_reserved":0,"allocated":8210694144,"data_stored":9910513929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":782639,"internal_metadata":1330187985},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,2,6,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:00:19 2023","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.35099999999999998,"15min":0.379},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.16900000000000001},"max":{"1min":0.56599999999999995,"5min":2.1600000000000001,"15min":4.2140000000000004},"last":0.251},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.505,"15min":0.48199999999999998},"min":{"1min":0.28000000000000003,"5min":0.218,"15min":0.188},"max":{"1min":0.78200000000000003,"5min":5.3280000000000003,"15min":5.3280000000000003},"last":0.40500000000000003}]},{"osd":1,"last update":"Tue Oct 17 02:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.52900000000000003,"15min":0.49099999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.19400000000000001},"max":{"1min":0.46899999999999997,"5min":9.5429999999999993,"15min":9.5429999999999993},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.57099999999999995,"15min":0.55000000000000004},"min":{"1min":0.26200000000000001,"5min":0.23100000000000001,"15min":0.19500000000000001},"max":{"1min":0.503,"5min":9.5999999999999996,"15min":9.5999999999999996},"last":0.42999999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739319,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5281572,"kb_used_data":3950620,"kb_used_omap":362,"kb_used_meta":1330581,"kb_avail":89090268,"statfs":{"total":96636764160,"available":91228434432,"internally_reserved":0,"allocated":4045434880,"data_stored":4883028264,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":371063,"internal_metadata":1362515593},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,7,3,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:01:12 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.36199999999999999,"15min":0.52800000000000002},"min":{"1min":0.16600000000000001,"5min":0.16300000000000001,"15min":0.16200000000000001},"max":{"1min":1.429,"5min":2.4809999999999999,"15min":23.177},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.41199999999999998,"15min":0.57999999999999996},"min":{"1min":0.23000000000000001,"5min":0.22,"15min":0.216},"max":{"1min":1.464,"5min":2.512,"15min":23.207999999999998},"last":0.27100000000000002}]},{"osd":2,"last update":"Tue Oct 17 02:01:12 2023","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.56799999999999995,"15min":0.50600000000000001},"min":{"1min":0.222,"5min":0.19700000000000001,"15min":0.188},"max":{"1min":6.8230000000000004,"5min":6.8230000000000004,"15min":6.8230000000000004},"last":0.222},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.52900000000000003,"15min":0.503},"min":{"1min":0.247,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":6.7800000000000002,"5min":6.7800000000000002,"15min":6.7800000000000002},"last":0.29499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739319,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5711248,"kb_used_data":4073416,"kb_used_omap":416,"kb_used_meta":1637407,"kb_avail":88660592,"statfs":{"total":96636764160,"available":90788446208,"internally_reserved":0,"allocated":4171177984,"data_stored":5033616879,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":426086,"internal_metadata":1676705690},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,4,7,13,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.436,"15min":0.44500000000000001},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.16200000000000001},"max":{"1min":4.101,"5min":4.101,"15min":5.9829999999999997},"last":0.17999999999999999},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.40799999999999997,"15min":0.48599999999999999},"min":{"1min":0.20499999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.2970000000000002,"5min":3.1429999999999998,"15min":6.0750000000000002},"last":0.63300000000000001}]},{"osd":2,"last update":"Tue Oct 17 02:00:56 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.47299999999999998,"15min":0.38800000000000001},"min":{"1min":0.20699999999999999,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":3.5750000000000002,"5min":3.5750000000000002,"15min":3.5750000000000002},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.64100000000000001,"15min":0.60499999999999998},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.19400000000000001},"max":{"1min":8.0169999999999995,"5min":8.827,"15min":10.378},"last":0.69299999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4170170368,"data_stored":5032767488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4044763136,"data_stored":4882595859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8211017728,"data_stored":9911447571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:01:22.702 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:01:22.703 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:01:22.703 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:01:22.703 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:01:22.704 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:01:42.706 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:01:43.288 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:01:43.289 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:01:43.344 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2491,"stamp":"2023-10-17T02:01:41.490149+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27892321112,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2037935,"num_read_kb":7953453,"num_write":2013264,"num_write_kb":8052707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87049,"ondisk_log_size":87049,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20533912,"kb_used_data":16098752,"kb_used_omap":1542,"kb_used_meta":4433593,"kb_avail":262581608,"statfs":{"total":289910292480,"available":268883566592,"internally_reserved":0,"allocated":16485122048,"data_stored":19884974112,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1579788,"internal_metadata":4539999476},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,7,11,17,20,10,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21766144,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5478,"num_read_kb":21860,"num_write":5300,"num_write_kb":21200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002085"},"pg_stats":[{"pgid":"2.7","version":"106'13233","reported_seq":18719,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-17T02:01:39.584436+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:01:39.584436+0000","last_peered":"2023-10-17T02:01:39.584436+0000","last_clean":"2023-10-17T02:01:39.584436+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:01:39.584436+0000","last_undegraded":"2023-10-17T02:01:39.584436+0000","last_fullsized":"2023-10-17T02:01:39.584436+0000","mapping_epoch":13,"log_start":"71'3200","ondisk_log_start":"71'3200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"106'9970","reported_seq":15258,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-17T02:01:39.584548+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:01:39.584548+0000","last_peered":"2023-10-17T02:01:39.584548+0000","last_clean":"2023-10-17T02:01:39.584548+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:01:39.584548+0000","last_undegraded":"2023-10-17T02:01:39.584548+0000","last_fullsized":"2023-10-17T02:01:39.584548+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":9970,"log_dups_size":0,"ondisk_log_size":9970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"106'6684","reported_seq":9514,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-17T02:01:39.583811+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:01:39.583811+0000","last_peered":"2023-10-17T02:01:39.583811+0000","last_clean":"2023-10-17T02:01:39.583811+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:01:39.583811+0000","last_undegraded":"2023-10-17T02:01:39.583811+0000","last_fullsized":"2023-10-17T02:01:39.583811+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6684,"log_dups_size":0,"ondisk_log_size":6684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"106'10003","reported_seq":14699,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-17T02:01:39.583987+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:01:39.583987+0000","last_peered":"2023-10-17T02:01:39.583987+0000","last_clean":"2023-10-17T02:01:39.583987+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:01:39.583987+0000","last_undegraded":"2023-10-17T02:01:39.583987+0000","last_fullsized":"2023-10-17T02:01:39.583987+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"106'512716","reported_seq":1021950,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:01:41.040602+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T02:01:41.040602+0000","last_peered":"2023-10-17T02:01:41.040602+0000","last_clean":"2023-10-17T02:01:41.040602+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:01:41.040602+0000","last_undegraded":"2023-10-17T02:01:41.040602+0000","last_fullsized":"2023-10-17T02:01:41.040602+0000","mapping_epoch":16,"log_start":"105'502710","ondisk_log_start":"105'502710","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2032,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 807s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7070822400,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":510018,"num_read_kb":2008144,"num_write":510275,"num_write_kb":2041100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":261,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-17T01:59:45.085795+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T01:59:45.085795+0000","last_peered":"2023-10-17T01:59:45.085795+0000","last_clean":"2023-10-17T01:59:45.085795+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T01:59:45.085795+0000","last_undegraded":"2023-10-17T01:59:45.085795+0000","last_fullsized":"2023-10-17T01:59:45.085795+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"106'493431","reported_seq":984150,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-17T02:01:41.091131+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:01:41.091131+0000","last_peered":"2023-10-17T02:01:41.091131+0000","last_clean":"2023-10-17T02:01:41.091131+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:01:41.091131+0000","last_undegraded":"2023-10-17T02:01:41.091131+0000","last_fullsized":"2023-10-17T02:01:41.091131+0000","mapping_epoch":16,"log_start":"105'483419","ondisk_log_start":"105'483419","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6798483456,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":491408,"num_read_kb":1936760,"num_write":490781,"num_write_kb":1963124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2790,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-17T02:01:34.463357+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:01:34.463357+0000","last_peered":"2023-10-17T02:01:34.463357+0000","last_clean":"2023-10-17T02:01:34.463357+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:01:34.463357+0000","last_undegraded":"2023-10-17T02:01:34.463357+0000","last_fullsized":"2023-10-17T02:01:34.463357+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"106'514036","reported_seq":1024487,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:01:41.037932+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T02:01:41.037932+0000","last_peered":"2023-10-17T02:01:41.037932+0000","last_clean":"2023-10-17T02:01:41.037932+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:01:41.037932+0000","last_undegraded":"2023-10-17T02:01:41.037932+0000","last_fullsized":"2023-10-17T02:01:41.037932+0000","mapping_epoch":16,"log_start":"105'504025","ondisk_log_start":"105'504025","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2356,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 815s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7088017408,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":511213,"num_read_kb":2013656,"num_write":511420,"num_write_kb":2045680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4186,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-17T02:01:34.506928+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:01:34.506928+0000","last_peered":"2023-10-17T02:01:34.506928+0000","last_clean":"2023-10-17T02:01:34.506928+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:01:34.506928+0000","last_undegraded":"2023-10-17T02:01:34.506928+0000","last_fullsized":"2023-10-17T02:01:34.506928+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"106'501445","reported_seq":1001941,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:01:41.107863+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:01:41.107863+0000","last_peered":"2023-10-17T02:01:41.107863+0000","last_clean":"2023-10-17T02:01:41.107863+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:01:41.107863+0000","last_undegraded":"2023-10-17T02:01:41.107863+0000","last_fullsized":"2023-10-17T02:01:41.107863+0000","mapping_epoch":16,"log_start":"105'491416","ondisk_log_start":"105'491416","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":26,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 15s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6934405139,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":501334,"num_read_kb":1974240,"num_write":500158,"num_write_kb":2000626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"106'9974","reported_seq":14034,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-17T02:01:39.583962+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:01:39.583962+0000","last_peered":"2023-10-17T02:01:39.583962+0000","last_clean":"2023-10-17T02:01:39.583962+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:01:39.583962+0000","last_undegraded":"2023-10-17T02:01:39.583962+0000","last_fullsized":"2023-10-17T02:01:39.583962+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":9974,"log_dups_size":0,"ondisk_log_size":9974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":282,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-17T02:00:37.357112+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:00:37.357112+0000","last_peered":"2023-10-17T02:00:37.357112+0000","last_clean":"2023-10-17T02:00:37.357112+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:00:37.357112+0000","last_undegraded":"2023-10-17T02:00:37.357112+0000","last_fullsized":"2023-10-17T02:00:37.357112+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27891728403,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2013973,"num_read_kb":7932800,"num_write":2012634,"num_write_kb":8050530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16483569664,"data_stored":19884429350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40058,"ondisk_log_size":40058,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":46892,"ondisk_log_size":46892,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739326,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9472464,"kb_used_data":8046472,"kb_used_omap":764,"kb_used_meta":1425219,"kb_avail":84899376,"statfs":{"total":96636764160,"available":86936961024,"internally_reserved":0,"allocated":8239587328,"data_stored":9939407113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":782639,"internal_metadata":1459424977},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,4,6,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:01:21 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34399999999999997,"15min":0.38300000000000001},"min":{"1min":0.192,"5min":0.17799999999999999,"15min":0.16900000000000001},"max":{"1min":1.5609999999999999,"5min":2.1600000000000001,"15min":4.2140000000000004},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.435,"5min":0.437,"15min":0.48999999999999999},"min":{"1min":0.23599999999999999,"5min":0.218,"15min":0.188},"max":{"1min":1.671,"5min":2.2050000000000001,"15min":5.3280000000000003},"last":0.57999999999999996}]},{"osd":1,"last update":"Tue Oct 17 02:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.52900000000000003,"15min":0.49099999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.19400000000000001},"max":{"1min":0.46899999999999997,"5min":9.5429999999999993,"15min":9.5429999999999993},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.57099999999999995,"15min":0.55000000000000004},"min":{"1min":0.26200000000000001,"5min":0.23100000000000001,"15min":0.19500000000000001},"max":{"1min":0.503,"5min":9.5999999999999996,"15min":9.5999999999999996},"last":0.66100000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739323,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5317716,"kb_used_data":3964428,"kb_used_omap":362,"kb_used_meta":1352917,"kb_avail":89054124,"statfs":{"total":96636764160,"available":91191422976,"internally_reserved":0,"allocated":4059574272,"data_stored":4897167656,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":371063,"internal_metadata":1385387657},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,6,2,0,10,9,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:01:12 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.36199999999999999,"15min":0.52800000000000002},"min":{"1min":0.16600000000000001,"5min":0.16300000000000001,"15min":0.16200000000000001},"max":{"1min":1.429,"5min":2.4809999999999999,"15min":23.177},"last":0.254},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.41199999999999998,"15min":0.57999999999999996},"min":{"1min":0.23000000000000001,"5min":0.22,"15min":0.216},"max":{"1min":1.464,"5min":2.512,"15min":23.207999999999998},"last":0.28499999999999998}]},{"osd":2,"last update":"Tue Oct 17 02:01:12 2023","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.56799999999999995,"15min":0.50600000000000001},"min":{"1min":0.222,"5min":0.19700000000000001,"15min":0.188},"max":{"1min":6.8230000000000004,"5min":6.8230000000000004,"15min":6.8230000000000004},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.52900000000000003,"15min":0.503},"min":{"1min":0.247,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":6.7800000000000002,"5min":6.7800000000000002,"15min":6.7800000000000002},"last":0.22600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739323,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5743732,"kb_used_data":4087852,"kb_used_omap":416,"kb_used_meta":1655455,"kb_avail":88628108,"statfs":{"total":96636764160,"available":90755182592,"internally_reserved":0,"allocated":4185960448,"data_stored":5048399343,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":426086,"internal_metadata":1695186842},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,11,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.436,"15min":0.44500000000000001},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.16200000000000001},"max":{"1min":4.101,"5min":4.101,"15min":5.9829999999999997},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.40799999999999997,"15min":0.48599999999999999},"min":{"1min":0.20499999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.2970000000000002,"5min":3.1429999999999998,"15min":6.0750000000000002},"last":0.20999999999999999}]},{"osd":2,"last update":"Tue Oct 17 02:00:56 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.47299999999999998,"15min":0.38800000000000001},"min":{"1min":0.20699999999999999,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":3.5750000000000002,"5min":3.5750000000000002,"15min":3.5750000000000002},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.64100000000000001,"15min":0.60499999999999998},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.19400000000000001},"max":{"1min":8.0169999999999995,"5min":8.827,"15min":10.378},"last":0.24399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4184862720,"data_stored":5047459840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4058890240,"data_stored":4896722963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8239816704,"data_stored":9940246547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:01:43.346 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:01:43.346 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:01:43.347 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:01:43.347 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:01:49.315 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:01:49.307+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T02:01:49.315 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:01:49.307+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T02:01:49.333 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:01:49.323+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbda750 2023-10-17T02:01:49.334 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:01:49.323+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T02:01:49.334 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:01:49.323+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T02:01:49.335 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:01:49.323+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T02:01:49.335 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:01:49.323+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T02:01:49.335 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:01:49.323+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T02:01:49.335 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:01:49.323+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T02:01:49.336 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:01:49.323+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T02:01:49.336 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:01:49.323+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T02:01:49.336 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:01:49.323+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T02:01:49.337 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:01:49.323+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T02:01:49.337 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:01:49.323+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T02:01:49.337 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:01:49.323+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T02:02:03.348 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:02:03.891 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:02:03.891 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:02:03.923 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2503,"stamp":"2023-10-17T02:02:03.493879+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27920280408,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2045082,"num_read_kb":7981937,"num_write":2020082,"num_write_kb":8079979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87373,"ondisk_log_size":87373,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20699812,"kb_used_data":16153224,"kb_used_omap":1542,"kb_used_meta":4545017,"kb_avail":262415708,"statfs":{"total":289910292480,"available":268713684992,"internally_reserved":0,"allocated":16540901376,"data_stored":19940740495,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1579788,"internal_metadata":4654097652},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,4,9,14,12,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14143488,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3672,"num_read_kb":14628,"num_write":3456,"num_write_kb":13824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001592"},"pg_stats":[{"pgid":"2.7","version":"107'13281","reported_seq":18783,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-10-17T02:02:00.064431+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:02:00.064431+0000","last_peered":"2023-10-17T02:02:00.064431+0000","last_clean":"2023-10-17T02:02:00.064431+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:02:00.064431+0000","last_undegraded":"2023-10-17T02:02:00.064431+0000","last_fullsized":"2023-10-17T02:02:00.064431+0000","mapping_epoch":13,"log_start":"71'3200","ondisk_log_start":"71'3200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"107'10006","reported_seq":15306,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-10-17T02:02:00.063944+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:02:00.063944+0000","last_peered":"2023-10-17T02:02:00.063944+0000","last_clean":"2023-10-17T02:02:00.063944+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:02:00.063944+0000","last_undegraded":"2023-10-17T02:02:00.063944+0000","last_fullsized":"2023-10-17T02:02:00.063944+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"107'6708","reported_seq":9548,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-10-17T02:02:00.063242+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:02:00.063242+0000","last_peered":"2023-10-17T02:02:00.063242+0000","last_clean":"2023-10-17T02:02:00.063242+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:02:00.063242+0000","last_undegraded":"2023-10-17T02:02:00.063242+0000","last_fullsized":"2023-10-17T02:02:00.063242+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6708,"log_dups_size":0,"ondisk_log_size":6708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"107'10039","reported_seq":14748,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-10-17T02:02:00.063477+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:02:00.063477+0000","last_peered":"2023-10-17T02:02:00.063477+0000","last_clean":"2023-10-17T02:02:00.063477+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:02:00.063477+0000","last_undegraded":"2023-10-17T02:02:00.063477+0000","last_fullsized":"2023-10-17T02:02:00.063477+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"107'514464","reported_seq":1025479,"reported_epoch":107,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:02:01.195014+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T02:02:01.195014+0000","last_peered":"2023-10-17T02:02:01.195014+0000","last_clean":"2023-10-17T02:02:01.195014+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:02:01.195014+0000","last_undegraded":"2023-10-17T02:02:01.195014+0000","last_fullsized":"2023-10-17T02:02:01.195014+0000","mapping_epoch":16,"log_start":"105'504410","ondisk_log_start":"105'504410","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2120,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 828s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7077773312,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":511790,"num_read_kb":2015192,"num_write":511972,"num_write_kb":2047888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":263,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-17T02:01:51.093692+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:01:51.093692+0000","last_peered":"2023-10-17T02:01:51.093692+0000","last_clean":"2023-10-17T02:01:51.093692+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:01:51.093692+0000","last_undegraded":"2023-10-17T02:01:51.093692+0000","last_fullsized":"2023-10-17T02:01:51.093692+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"107'495035","reported_seq":987477,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-10-17T02:02:01.179464+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:02:01.179464+0000","last_peered":"2023-10-17T02:02:01.179464+0000","last_clean":"2023-10-17T02:02:01.179464+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:02:01.179464+0000","last_undegraded":"2023-10-17T02:02:01.179464+0000","last_fullsized":"2023-10-17T02:02:01.179464+0000","mapping_epoch":16,"log_start":"105'485019","ondisk_log_start":"105'485019","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6805041152,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":493129,"num_read_kb":1943620,"num_write":492385,"num_write_kb":1969540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2800,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-17T02:01:54.946225+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:01:54.946225+0000","last_peered":"2023-10-17T02:01:54.946225+0000","last_clean":"2023-10-17T02:01:54.946225+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:01:54.946225+0000","last_undegraded":"2023-10-17T02:01:54.946225+0000","last_fullsized":"2023-10-17T02:01:54.946225+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"107'515873","reported_seq":1028175,"reported_epoch":107,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:02:01.167340+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T02:02:01.167340+0000","last_peered":"2023-10-17T02:02:01.167340+0000","last_clean":"2023-10-17T02:02:01.167340+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:02:01.167340+0000","last_undegraded":"2023-10-17T02:02:01.167340+0000","last_fullsized":"2023-10-17T02:02:01.167340+0000","mapping_epoch":16,"log_start":"105'505825","ondisk_log_start":"105'505825","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2430,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 836s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7095418880,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":513053,"num_read_kb":2020984,"num_write":513217,"num_write_kb":2052868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4199,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-17T02:01:54.946611+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:01:54.946611+0000","last_peered":"2023-10-17T02:01:54.946611+0000","last_clean":"2023-10-17T02:01:54.946611+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:01:54.946611+0000","last_undegraded":"2023-10-17T02:01:54.946611+0000","last_fullsized":"2023-10-17T02:01:54.946611+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"107'503200","reported_seq":1005514,"reported_epoch":107,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:02:01.206662+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:02:01.206662+0000","last_peered":"2023-10-17T02:02:01.206662+0000","last_clean":"2023-10-17T02:02:01.206662+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:02:01.206662+0000","last_undegraded":"2023-10-17T02:02:01.206662+0000","last_fullsized":"2023-10-17T02:02:01.206662+0000","mapping_epoch":16,"log_start":"105'493116","ondisk_log_start":"105'493116","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":90,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 35s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6941454355,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":503148,"num_read_kb":1981488,"num_write":501878,"num_write_kb":2007506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"107'10010","reported_seq":14084,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-10-17T02:02:00.063505+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:02:00.063505+0000","last_peered":"2023-10-17T02:02:00.063505+0000","last_clean":"2023-10-17T02:02:00.063505+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:02:00.063505+0000","last_undegraded":"2023-10-17T02:02:00.063505+0000","last_fullsized":"2023-10-17T02:02:00.063505+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":283,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-17T02:00:40.764303+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:00:40.764303+0000","last_peered":"2023-10-17T02:00:40.764303+0000","last_clean":"2023-10-17T02:00:40.764303+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:00:40.764303+0000","last_undegraded":"2023-10-17T02:00:40.764303+0000","last_fullsized":"2023-10-17T02:00:40.764303+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27919687699,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2021120,"num_read_kb":7961284,"num_write":2019452,"num_write_kb":8077802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16539508736,"data_stored":19940368422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40202,"ondisk_log_size":40202,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":47072,"ondisk_log_size":47072,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739330,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9487448,"kb_used_data":8073868,"kb_used_omap":764,"kb_used_meta":1412803,"kb_avail":84884392,"statfs":{"total":96636764160,"available":86921617408,"internally_reserved":0,"allocated":8267640832,"data_stored":9967456302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":782639,"internal_metadata":1446710993},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:01:21 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34399999999999997,"15min":0.38300000000000001},"min":{"1min":0.192,"5min":0.17799999999999999,"15min":0.16900000000000001},"max":{"1min":1.5609999999999999,"5min":2.1600000000000001,"15min":4.2140000000000004},"last":2.1099999999999999},{"interface":"front","average":{"1min":0.435,"5min":0.437,"15min":0.48999999999999999},"min":{"1min":0.23599999999999999,"5min":0.218,"15min":0.188},"max":{"1min":1.671,"5min":2.2050000000000001,"15min":5.3280000000000003},"last":0.21099999999999999}]},{"osd":1,"last update":"Tue Oct 17 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.44,"15min":0.48499999999999999},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.20300000000000001},"max":{"1min":0.63400000000000001,"5min":5.2169999999999996,"15min":9.5429999999999993},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.377,"5min":0.48299999999999998,"15min":0.54700000000000004},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.19500000000000001},"max":{"1min":0.69799999999999995,"5min":5.4379999999999997,"15min":9.5999999999999996},"last":0.32400000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739327,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5397548,"kb_used_data":3977636,"kb_used_omap":362,"kb_used_meta":1419541,"kb_avail":88974292,"statfs":{"total":96636764160,"available":91109675008,"internally_reserved":0,"allocated":4073099264,"data_stored":4910688333,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":371063,"internal_metadata":1453610633},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,5,6,1,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:01:12 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.36199999999999999,"15min":0.52800000000000002},"min":{"1min":0.16600000000000001,"5min":0.16300000000000001,"15min":0.16200000000000001},"max":{"1min":1.429,"5min":2.4809999999999999,"15min":23.177},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.41199999999999998,"15min":0.57999999999999996},"min":{"1min":0.23000000000000001,"5min":0.22,"15min":0.216},"max":{"1min":1.464,"5min":2.512,"15min":23.207999999999998},"last":0.46800000000000003}]},{"osd":2,"last update":"Tue Oct 17 02:01:12 2023","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.56799999999999995,"15min":0.50600000000000001},"min":{"1min":0.222,"5min":0.19700000000000001,"15min":0.188},"max":{"1min":6.8230000000000004,"5min":6.8230000000000004,"15min":6.8230000000000004},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.52900000000000003,"15min":0.503},"min":{"1min":0.247,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":6.7800000000000002,"5min":6.7800000000000002,"15min":6.7800000000000002},"last":0.36399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739327,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5814816,"kb_used_data":4101720,"kb_used_omap":416,"kb_used_meta":1712671,"kb_avail":88557024,"statfs":{"total":96636764160,"available":90682392576,"internally_reserved":0,"allocated":4200161280,"data_stored":5062595860,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":426086,"internal_metadata":1753776026},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,6,9,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.45100000000000001,"15min":0.44500000000000001},"min":{"1min":0.17999999999999999,"5min":0.17000000000000001,"15min":0.16200000000000001},"max":{"1min":2.3889999999999998,"5min":4.101,"15min":5.9829999999999997},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.40100000000000002,"15min":0.48399999999999999},"min":{"1min":0.20599999999999999,"5min":0.191,"15min":0.191},"max":{"1min":0.63300000000000001,"5min":3.1429999999999998,"15min":6.0750000000000002},"last":0.27600000000000002}]},{"osd":2,"last update":"Tue Oct 17 02:01:58 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50600000000000001,"15min":0.40500000000000003},"min":{"1min":0.22900000000000001,"5min":0.20699999999999999,"15min":0.191},"max":{"1min":6.0010000000000003,"5min":6.0010000000000003,"15min":6.0010000000000003},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.59599999999999997,"15min":0.59999999999999998},"min":{"1min":0.24299999999999999,"5min":0.20499999999999999,"15min":0.19400000000000001},"max":{"1min":0.69299999999999995,"5min":8.827,"15min":10.378},"last":0.33300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4199178240,"data_stored":5061775360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4072501248,"data_stored":4910333971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8267829248,"data_stored":9968259091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:02:03.925 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:02:03.926 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:02:03.926 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:02:03.926 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:02:23.928 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:02:24.765 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:02:24.766 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:02:24.819 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2513,"stamp":"2023-10-17T02:02:23.497251+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27949738840,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2052447,"num_read_kb":8011237,"num_write":2027286,"num_write_kb":8108795,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87485,"ondisk_log_size":87485,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20909596,"kb_used_data":16210300,"kb_used_omap":1542,"kb_used_meta":4697721,"kb_avail":262205924,"statfs":{"total":289910292480,"available":268498866176,"internally_reserved":0,"allocated":16599347200,"data_stored":19999186319,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1579788,"internal_metadata":4810466548},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,5,10,12,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14675968,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3621,"num_read_kb":14412,"num_write":3592,"num_write_kb":14368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001982"},"pg_stats":[{"pgid":"2.7","version":"107'13337","reported_seq":18855,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-10-17T02:02:20.543551+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:02:20.543551+0000","last_peered":"2023-10-17T02:02:20.543551+0000","last_clean":"2023-10-17T02:02:20.543551+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:02:20.543551+0000","last_undegraded":"2023-10-17T02:02:20.543551+0000","last_fullsized":"2023-10-17T02:02:20.543551+0000","mapping_epoch":13,"log_start":"71'3300","ondisk_log_start":"71'3300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"107'10048","reported_seq":15360,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-10-17T02:02:20.544561+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:02:20.544561+0000","last_peered":"2023-10-17T02:02:20.544561+0000","last_clean":"2023-10-17T02:02:20.544561+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:02:20.544561+0000","last_undegraded":"2023-10-17T02:02:20.544561+0000","last_fullsized":"2023-10-17T02:02:20.544561+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"107'6736","reported_seq":9584,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-10-17T02:02:20.543268+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:02:20.543268+0000","last_peered":"2023-10-17T02:02:20.543268+0000","last_clean":"2023-10-17T02:02:20.543268+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:02:20.543268+0000","last_undegraded":"2023-10-17T02:02:20.543268+0000","last_fullsized":"2023-10-17T02:02:20.543268+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6736,"log_dups_size":0,"ondisk_log_size":6736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"107'10081","reported_seq":14802,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-10-17T02:02:20.543491+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:02:20.543491+0000","last_peered":"2023-10-17T02:02:20.543491+0000","last_clean":"2023-10-17T02:02:20.543491+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:02:20.543491+0000","last_undegraded":"2023-10-17T02:02:20.543491+0000","last_fullsized":"2023-10-17T02:02:20.543491+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"107'516294","reported_seq":1029161,"reported_epoch":107,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:02:21.258621+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T02:02:21.258621+0000","last_peered":"2023-10-17T02:02:21.258621+0000","last_clean":"2023-10-17T02:02:21.258621+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:02:21.258621+0000","last_undegraded":"2023-10-17T02:02:21.258621+0000","last_fullsized":"2023-10-17T02:02:21.258621+0000","mapping_epoch":16,"log_start":"105'506210","ondisk_log_start":"105'506210","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2168,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 848s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7085133824,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":513634,"num_read_kb":2022524,"num_write":513776,"num_write_kb":2055104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":263,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-17T02:01:51.093692+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:01:51.093692+0000","last_peered":"2023-10-17T02:01:51.093692+0000","last_clean":"2023-10-17T02:01:51.093692+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:01:51.093692+0000","last_undegraded":"2023-10-17T02:01:51.093692+0000","last_fullsized":"2023-10-17T02:01:51.093692+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"107'496838","reported_seq":991048,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-10-17T02:02:21.379998+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:02:21.379998+0000","last_peered":"2023-10-17T02:02:21.379998+0000","last_clean":"2023-10-17T02:02:21.379998+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:02:21.379998+0000","last_undegraded":"2023-10-17T02:02:21.379998+0000","last_fullsized":"2023-10-17T02:02:21.379998+0000","mapping_epoch":16,"log_start":"105'486819","ondisk_log_start":"105'486819","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6812389376,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":494898,"num_read_kb":1950676,"num_write":494188,"num_write_kb":1976752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2808,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-10-17T02:02:15.427284+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:02:15.427284+0000","last_peered":"2023-10-17T02:02:15.427284+0000","last_clean":"2023-10-17T02:02:15.427284+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:02:15.427284+0000","last_undegraded":"2023-10-17T02:02:15.427284+0000","last_fullsized":"2023-10-17T02:02:15.427284+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"107'517741","reported_seq":1031889,"reported_epoch":107,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:02:21.259224+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T02:02:21.259224+0000","last_peered":"2023-10-17T02:02:21.259224+0000","last_clean":"2023-10-17T02:02:21.259224+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:02:21.259224+0000","last_undegraded":"2023-10-17T02:02:21.259224+0000","last_fullsized":"2023-10-17T02:02:21.259224+0000","mapping_epoch":16,"log_start":"105'507725","ondisk_log_start":"105'507725","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2490,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 856s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7102930944,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":514889,"num_read_kb":2028288,"num_write":515052,"num_write_kb":2060208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4211,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-10-17T02:02:15.498987+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:02:15.498987+0000","last_peered":"2023-10-17T02:02:15.498987+0000","last_clean":"2023-10-17T02:02:15.498987+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:02:15.498987+0000","last_undegraded":"2023-10-17T02:02:15.498987+0000","last_fullsized":"2023-10-17T02:02:15.498987+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"107'505001","reported_seq":1009250,"reported_epoch":107,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:02:21.378844+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:02:21.378844+0000","last_peered":"2023-10-17T02:02:21.378844+0000","last_clean":"2023-10-17T02:02:21.378844+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:02:21.378844+0000","last_undegraded":"2023-10-17T02:02:21.378844+0000","last_fullsized":"2023-10-17T02:02:21.378844+0000","mapping_epoch":16,"log_start":"105'494916","ondisk_log_start":"105'494916","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":158,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 55s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6948691987,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":505064,"num_read_kb":1989096,"num_write":503640,"num_write_kb":2014554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"107'10052","reported_seq":14138,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-10-17T02:02:20.543288+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:02:20.543288+0000","last_peered":"2023-10-17T02:02:20.543288+0000","last_clean":"2023-10-17T02:02:20.543288+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:02:20.543288+0000","last_undegraded":"2023-10-17T02:02:20.543288+0000","last_fullsized":"2023-10-17T02:02:20.543288+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":283,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-17T02:00:40.764303+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:00:40.764303+0000","last_peered":"2023-10-17T02:00:40.764303+0000","last_clean":"2023-10-17T02:00:40.764303+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:00:40.764303+0000","last_undegraded":"2023-10-17T02:00:40.764303+0000","last_fullsized":"2023-10-17T02:00:40.764303+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27949146131,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2028485,"num_read_kb":7990584,"num_write":2026656,"num_write_kb":8106618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16598433792,"data_stored":19999293478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40204,"ondisk_log_size":40204,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":47182,"ondisk_log_size":47182,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739334,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9596408,"kb_used_data":8102312,"kb_used_omap":764,"kb_used_meta":1493315,"kb_avail":84775432,"statfs":{"total":96636764160,"available":86810042368,"internally_reserved":0,"allocated":8296767488,"data_stored":9996582958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":782639,"internal_metadata":1529155281},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:01:21 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34399999999999997,"15min":0.38300000000000001},"min":{"1min":0.192,"5min":0.17799999999999999,"15min":0.16900000000000001},"max":{"1min":1.5609999999999999,"5min":2.1600000000000001,"15min":4.2140000000000004},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.435,"5min":0.437,"15min":0.48999999999999999},"min":{"1min":0.23599999999999999,"5min":0.218,"15min":0.188},"max":{"1min":1.671,"5min":2.2050000000000001,"15min":5.3280000000000003},"last":0.42699999999999999}]},{"osd":1,"last update":"Tue Oct 17 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.44,"15min":0.48499999999999999},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.20300000000000001},"max":{"1min":0.63400000000000001,"5min":5.2169999999999996,"15min":9.5429999999999993},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.377,"5min":0.48299999999999998,"15min":0.54700000000000004},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.19500000000000001},"max":{"1min":0.69799999999999995,"5min":5.4379999999999997,"15min":9.5999999999999996},"last":0.27300000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739331,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5447840,"kb_used_data":3991832,"kb_used_omap":362,"kb_used_meta":1455637,"kb_avail":88924000,"statfs":{"total":96636764160,"available":91058176000,"internally_reserved":0,"allocated":4087635968,"data_stored":4925225037,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":371063,"internal_metadata":1490572937},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,7,2,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:02:13 2023","interfaces":[{"interface":"back","average":{"1min":0.28399999999999997,"5min":0.33900000000000002,"15min":0.52600000000000002},"min":{"1min":0.193,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.42699999999999999,"5min":2.4809999999999999,"15min":23.177},"last":5.5869999999999997},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.38800000000000001,"15min":0.57799999999999996},"min":{"1min":0.249,"5min":0.22,"15min":0.22},"max":{"1min":0.46800000000000003,"5min":2.512,"15min":23.207999999999998},"last":5.6360000000000001}]},{"osd":2,"last update":"Tue Oct 17 02:02:13 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53600000000000003,"15min":0.50700000000000001},"min":{"1min":0.217,"5min":0.19700000000000001,"15min":0.188},"max":{"1min":4.4429999999999996,"5min":6.8230000000000004,"15min":6.8230000000000004},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.30399999999999999,"5min":0.441,"15min":0.48099999999999998},"min":{"1min":0.22600000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.38,"5min":6.7800000000000002,"15min":6.7800000000000002},"last":0.48099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739331,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5865348,"kb_used_data":4116156,"kb_used_omap":416,"kb_used_meta":1748767,"kb_avail":88506492,"statfs":{"total":96636764160,"available":90630647808,"internally_reserved":0,"allocated":4214943744,"data_stored":5077378324,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":426086,"internal_metadata":1790738330},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,3,1,8,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.45100000000000001,"15min":0.44500000000000001},"min":{"1min":0.17999999999999999,"5min":0.17000000000000001,"15min":0.16200000000000001},"max":{"1min":2.3889999999999998,"5min":4.101,"15min":5.9829999999999997},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.40100000000000002,"15min":0.48399999999999999},"min":{"1min":0.20599999999999999,"5min":0.191,"15min":0.191},"max":{"1min":0.63300000000000001,"5min":3.1429999999999998,"15min":6.0750000000000002},"last":0.23899999999999999}]},{"osd":2,"last update":"Tue Oct 17 02:01:58 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50600000000000001,"15min":0.40500000000000003},"min":{"1min":0.22900000000000001,"5min":0.20699999999999999,"15min":0.191},"max":{"1min":6.0010000000000003,"5min":6.0010000000000003,"15min":6.0010000000000003},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.59599999999999997,"15min":0.59999999999999998},"min":{"1min":0.24299999999999999,"5min":0.20499999999999999,"15min":0.19400000000000001},"max":{"1min":0.69299999999999995,"5min":8.827,"15min":10.378},"last":0.30099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4214079488,"data_stored":5076676608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4087099392,"data_stored":4924932115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8297254912,"data_stored":9997684755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:02:24.820 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:02:24.821 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:02:24.821 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:02:24.822 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:02:44.823 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:02:45.282 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:02:45.284 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:02:45.313 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2523,"stamp":"2023-10-17T02:02:43.500725+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27978267480,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2059497,"num_read_kb":8039281,"num_write":2034332,"num_write_kb":8136979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87353,"ondisk_log_size":87353,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21142724,"kb_used_data":16268520,"kb_used_omap":1542,"kb_used_meta":4872633,"kb_avail":261972796,"statfs":{"total":289910292480,"available":268260143104,"internally_reserved":0,"allocated":16658964480,"data_stored":20058803599,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1579788,"internal_metadata":4989576436},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,4,4,5,15,10,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14307328,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3463,"num_read_kb":13768,"num_write":3533,"num_write_kb":14132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002172"},"pg_stats":[{"pgid":"2.7","version":"107'13417","reported_seq":18951,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-10-17T02:02:41.023632+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:02:41.023632+0000","last_peered":"2023-10-17T02:02:41.023632+0000","last_clean":"2023-10-17T02:02:41.023632+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:02:41.023632+0000","last_undegraded":"2023-10-17T02:02:41.023632+0000","last_fullsized":"2023-10-17T02:02:41.023632+0000","mapping_epoch":13,"log_start":"73'3400","ondisk_log_start":"73'3400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"107'10108","reported_seq":15433,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-10-17T02:02:41.024509+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:02:41.024509+0000","last_peered":"2023-10-17T02:02:41.024509+0000","last_clean":"2023-10-17T02:02:41.024509+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:02:41.024509+0000","last_undegraded":"2023-10-17T02:02:41.024509+0000","last_fullsized":"2023-10-17T02:02:41.024509+0000","mapping_epoch":13,"log_start":"40'100","ondisk_log_start":"40'100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"107'6776","reported_seq":9632,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-10-17T02:02:41.023269+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:02:41.023269+0000","last_peered":"2023-10-17T02:02:41.023269+0000","last_clean":"2023-10-17T02:02:41.023269+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:02:41.023269+0000","last_undegraded":"2023-10-17T02:02:41.023269+0000","last_fullsized":"2023-10-17T02:02:41.023269+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6776,"log_dups_size":0,"ondisk_log_size":6776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"107'10141","reported_seq":14874,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-10-17T02:02:41.023437+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:02:41.023437+0000","last_peered":"2023-10-17T02:02:41.023437+0000","last_clean":"2023-10-17T02:02:41.023437+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:02:41.023437+0000","last_undegraded":"2023-10-17T02:02:41.023437+0000","last_fullsized":"2023-10-17T02:02:41.023437+0000","mapping_epoch":13,"log_start":"40'100","ondisk_log_start":"40'100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"107'518164","reported_seq":1032830,"reported_epoch":107,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:02:41.419203+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T02:02:41.419203+0000","last_peered":"2023-10-17T02:02:41.419203+0000","last_clean":"2023-10-17T02:02:41.419203+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:02:41.419203+0000","last_undegraded":"2023-10-17T02:02:41.419203+0000","last_fullsized":"2023-10-17T02:02:41.419203+0000","mapping_epoch":16,"log_start":"106'508110","ondisk_log_start":"106'508110","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2216,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 868s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7092588544,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":515422,"num_read_kb":2029628,"num_write":515619,"num_write_kb":2062476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":263,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-17T02:01:51.093692+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:01:51.093692+0000","last_peered":"2023-10-17T02:01:51.093692+0000","last_clean":"2023-10-17T02:01:51.093692+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:01:51.093692+0000","last_undegraded":"2023-10-17T02:01:51.093692+0000","last_fullsized":"2023-10-17T02:01:51.093692+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"107'498561","reported_seq":994523,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-10-17T02:02:41.503187+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:02:41.503187+0000","last_peered":"2023-10-17T02:02:41.503187+0000","last_clean":"2023-10-17T02:02:41.503187+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:02:41.503187+0000","last_undegraded":"2023-10-17T02:02:41.503187+0000","last_fullsized":"2023-10-17T02:02:41.503187+0000","mapping_epoch":16,"log_start":"106'488519","ondisk_log_start":"106'488519","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6819389440,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":496653,"num_read_kb":1957660,"num_write":495911,"num_write_kb":1983644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2816,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-10-17T02:02:35.909461+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:02:35.909461+0000","last_peered":"2023-10-17T02:02:35.909461+0000","last_clean":"2023-10-17T02:02:35.909461+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:02:35.909461+0000","last_undegraded":"2023-10-17T02:02:35.909461+0000","last_fullsized":"2023-10-17T02:02:35.909461+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"107'519562","reported_seq":1035522,"reported_epoch":107,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:02:41.403955+0000","last_change":"2023-10-17T01:48:04.887127+0000","last_active":"2023-10-17T02:02:41.403955+0000","last_peered":"2023-10-17T02:02:41.403955+0000","last_clean":"2023-10-17T02:02:41.403955+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:02:41.403955+0000","last_undegraded":"2023-10-17T02:02:41.403955+0000","last_fullsized":"2023-10-17T02:02:41.403955+0000","mapping_epoch":16,"log_start":"106'509525","ondisk_log_start":"106'509525","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2570,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 876s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7110115328,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":516691,"num_read_kb":2035468,"num_write":516829,"num_write_kb":2067316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4224,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-10-17T02:02:35.904157+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:02:35.904157+0000","last_peered":"2023-10-17T02:02:35.904157+0000","last_clean":"2023-10-17T02:02:35.904157+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:02:35.904157+0000","last_undegraded":"2023-10-17T02:02:35.904157+0000","last_fullsized":"2023-10-17T02:02:35.904157+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"107'506755","reported_seq":1012715,"reported_epoch":107,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:02:41.509618+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:02:41.509618+0000","last_peered":"2023-10-17T02:02:41.509618+0000","last_clean":"2023-10-17T02:02:41.509618+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:02:41.509618+0000","last_undegraded":"2023-10-17T02:02:41.509618+0000","last_fullsized":"2023-10-17T02:02:41.509618+0000","mapping_epoch":16,"log_start":"106'496716","ondisk_log_start":"106'496716","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":254,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 76s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6955581459,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":506769,"num_read_kb":1995872,"num_write":505343,"num_write_kb":2021366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"107'10112","reported_seq":14210,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-10-17T02:02:41.023669+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:02:41.023669+0000","last_peered":"2023-10-17T02:02:41.023669+0000","last_clean":"2023-10-17T02:02:41.023669+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:02:41.023669+0000","last_undegraded":"2023-10-17T02:02:41.023669+0000","last_fullsized":"2023-10-17T02:02:41.023669+0000","mapping_epoch":13,"log_start":"40'100","ondisk_log_start":"40'100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":284,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-10-17T02:01:57.317166+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:01:57.317166+0000","last_peered":"2023-10-17T02:01:57.317166+0000","last_clean":"2023-10-17T02:01:57.317166+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:01:57.317166+0000","last_undegraded":"2023-10-17T02:01:57.317166+0000","last_fullsized":"2023-10-17T02:01:57.317166+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27977674771,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2035535,"num_read_kb":8018628,"num_write":2033702,"num_write_kb":8134802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16656244736,"data_stored":20057104422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40172,"ondisk_log_size":40172,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":47082,"ondisk_log_size":47082,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739338,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9711932,"kb_used_data":8131760,"kb_used_omap":764,"kb_used_meta":1579395,"kb_avail":84659908,"statfs":{"total":96636764160,"available":86691745792,"internally_reserved":0,"allocated":8326922240,"data_stored":10026737710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":782639,"internal_metadata":1617301201},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:02:23 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.439,"15min":0.41699999999999998},"min":{"1min":0.18099999999999999,"5min":0.17799999999999999,"15min":0.16900000000000001},"max":{"1min":8.4190000000000005,"5min":8.4190000000000005,"15min":8.4190000000000005},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.749,"5min":0.497,"15min":0.5},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.188},"max":{"1min":8.3559999999999999,"5min":8.3559999999999999,"15min":8.3559999999999999},"last":0.42199999999999999}]},{"osd":1,"last update":"Tue Oct 17 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.44,"15min":0.48499999999999999},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.20300000000000001},"max":{"1min":0.63400000000000001,"5min":5.2169999999999996,"15min":9.5429999999999993},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.377,"5min":0.48299999999999998,"15min":0.54700000000000004},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.19500000000000001},"max":{"1min":0.69799999999999995,"5min":5.4379999999999997,"15min":9.5999999999999996},"last":0.51000000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739335,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5498304,"kb_used_data":4006200,"kb_used_omap":362,"kb_used_meta":1491733,"kb_avail":88873536,"statfs":{"total":96636764160,"available":91006500864,"internally_reserved":0,"allocated":4102348800,"data_stored":4939937869,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":371063,"internal_metadata":1527535241},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,2,1,2,5,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:02:13 2023","interfaces":[{"interface":"back","average":{"1min":0.28399999999999997,"5min":0.33900000000000002,"15min":0.52600000000000002},"min":{"1min":0.193,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.42699999999999999,"5min":2.4809999999999999,"15min":23.177},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.38800000000000001,"15min":0.57799999999999996},"min":{"1min":0.249,"5min":0.22,"15min":0.22},"max":{"1min":0.46800000000000003,"5min":2.512,"15min":23.207999999999998},"last":0.59899999999999998}]},{"osd":2,"last update":"Tue Oct 17 02:02:13 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53600000000000003,"15min":0.50700000000000001},"min":{"1min":0.217,"5min":0.19700000000000001,"15min":0.188},"max":{"1min":4.4429999999999996,"5min":6.8230000000000004,"15min":6.8230000000000004},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.30399999999999999,"5min":0.441,"15min":0.48099999999999998},"min":{"1min":0.22600000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.38,"5min":6.7800000000000002,"15min":6.7800000000000002},"last":0.65400000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739335,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5932488,"kb_used_data":4130560,"kb_used_omap":416,"kb_used_meta":1801503,"kb_avail":88439352,"statfs":{"total":96636764160,"available":90561896448,"internally_reserved":0,"allocated":4229693440,"data_stored":5092128020,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":426086,"internal_metadata":1844739994},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,2,8,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.45100000000000001,"15min":0.44500000000000001},"min":{"1min":0.17999999999999999,"5min":0.17000000000000001,"15min":0.16200000000000001},"max":{"1min":2.3889999999999998,"5min":4.101,"15min":5.9829999999999997},"last":0.37},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.40100000000000002,"15min":0.48399999999999999},"min":{"1min":0.20599999999999999,"5min":0.191,"15min":0.191},"max":{"1min":0.63300000000000001,"5min":3.1429999999999998,"15min":6.0750000000000002},"last":0.318}]},{"osd":2,"last update":"Tue Oct 17 02:01:58 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50600000000000001,"15min":0.40500000000000003},"min":{"1min":0.22900000000000001,"5min":0.20699999999999999,"15min":0.191},"max":{"1min":6.0010000000000003,"5min":6.0010000000000003,"15min":6.0010000000000003},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.59599999999999997,"15min":0.59999999999999998},"min":{"1min":0.24299999999999999,"5min":0.20499999999999999,"15min":0.19400000000000001},"max":{"1min":0.69299999999999995,"5min":8.827,"15min":10.378},"last":0.35099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4228915200,"data_stored":5091512320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4101136384,"data_stored":4938969107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8326193152,"data_stored":10026622995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:02:45.315 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:02:45.315 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:02:45.316 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:02:45.316 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:03:05.317 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:03:05.922 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:03:05.923 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:03:05.955 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2535,"stamp":"2023-10-17T02:03:05.506480+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28006951768,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2066686,"num_read_kb":8067853,"num_write":2041405,"num_write_kb":8165271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87462,"ondisk_log_size":87462,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21450788,"kb_used_data":16331596,"kb_used_omap":1542,"kb_used_meta":5117625,"kb_avail":261664732,"statfs":{"total":289910292480,"available":267944685568,"internally_reserved":0,"allocated":16723554304,"data_stored":20123380349,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1579788,"internal_metadata":5240448244},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,5,4,17,10,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14635008,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3584,"num_read_kb":14236,"num_write":3600,"num_write_kb":14400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002001"},"pg_stats":[{"pgid":"2.7","version":"108'13457","reported_seq":19009,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-17T02:03:01.506945+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:03:01.506945+0000","last_peered":"2023-10-17T02:03:01.506945+0000","last_clean":"2023-10-17T02:03:01.506945+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:03:01.506945+0000","last_undegraded":"2023-10-17T02:03:01.506945+0000","last_fullsized":"2023-10-17T02:03:01.506945+0000","mapping_epoch":13,"log_start":"73'3400","ondisk_log_start":"73'3400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"108'10138","reported_seq":15476,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-17T02:03:01.503589+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:03:01.503589+0000","last_peered":"2023-10-17T02:03:01.503589+0000","last_clean":"2023-10-17T02:03:01.503589+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:03:01.503589+0000","last_undegraded":"2023-10-17T02:03:01.503589+0000","last_fullsized":"2023-10-17T02:03:01.503589+0000","mapping_epoch":13,"log_start":"40'100","ondisk_log_start":"40'100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"108'6796","reported_seq":9661,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-17T02:03:01.503106+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:03:01.503106+0000","last_peered":"2023-10-17T02:03:01.503106+0000","last_clean":"2023-10-17T02:03:01.503106+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:03:01.503106+0000","last_undegraded":"2023-10-17T02:03:01.503106+0000","last_fullsized":"2023-10-17T02:03:01.503106+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6796,"log_dups_size":0,"ondisk_log_size":6796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"108'10171","reported_seq":14917,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-17T02:03:01.503628+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:03:01.503628+0000","last_peered":"2023-10-17T02:03:01.503628+0000","last_clean":"2023-10-17T02:03:01.503628+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:03:01.503628+0000","last_undegraded":"2023-10-17T02:03:01.503628+0000","last_fullsized":"2023-10-17T02:03:01.503628+0000","mapping_epoch":13,"log_start":"40'100","ondisk_log_start":"40'100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"108'520041","reported_seq":1036476,"reported_epoch":108,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:03:01.472658+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T02:03:01.472658+0000","last_peered":"2023-10-17T02:03:01.472658+0000","last_clean":"2023-10-17T02:03:01.472658+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:03:01.472658+0000","last_undegraded":"2023-10-17T02:03:01.472658+0000","last_fullsized":"2023-10-17T02:03:01.472658+0000","mapping_epoch":16,"log_start":"106'510010","ondisk_log_start":"106'510010","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2276,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 888s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7100067840,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":517180,"num_read_kb":2036620,"num_write":517459,"num_write_kb":2069836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":264,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-10-17T02:01:57.319523+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:01:57.319523+0000","last_peered":"2023-10-17T02:01:57.319523+0000","last_clean":"2023-10-17T02:01:57.319523+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:01:57.319523+0000","last_undegraded":"2023-10-17T02:01:57.319523+0000","last_fullsized":"2023-10-17T02:01:57.319523+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"108'500264","reported_seq":998020,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-17T02:03:01.655787+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:03:01.655787+0000","last_peered":"2023-10-17T02:03:01.655787+0000","last_clean":"2023-10-17T02:03:01.655787+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:03:01.655787+0000","last_undegraded":"2023-10-17T02:03:01.655787+0000","last_fullsized":"2023-10-17T02:03:01.655787+0000","mapping_epoch":16,"log_start":"106'490219","ondisk_log_start":"106'490219","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6826291200,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":498444,"num_read_kb":1964772,"num_write":497614,"num_write_kb":1990456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2827,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-17T02:03:01.503297+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:03:01.503297+0000","last_peered":"2023-10-17T02:03:01.503297+0000","last_clean":"2023-10-17T02:03:01.503297+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:03:01.503297+0000","last_undegraded":"2023-10-17T02:03:01.503297+0000","last_fullsized":"2023-10-17T02:03:01.503297+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"108'521376","reported_seq":1039205,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-17T02:03:01.479709+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:03:01.479709+0000","last_peered":"2023-10-17T02:03:01.479709+0000","last_clean":"2023-10-17T02:03:01.479709+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:03:01.479709+0000","last_undegraded":"2023-10-17T02:03:01.479709+0000","last_fullsized":"2023-10-17T02:03:01.479709+0000","mapping_epoch":16,"log_start":"106'511325","ondisk_log_start":"106'511325","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7117369344,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":518550,"num_read_kb":2042864,"num_write":518622,"num_write_kb":2074488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4240,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-17T02:03:01.580612+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:03:01.580612+0000","last_peered":"2023-10-17T02:03:01.580612+0000","last_clean":"2023-10-17T02:03:01.580612+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:03:01.580612+0000","last_undegraded":"2023-10-17T02:03:01.580612+0000","last_fullsized":"2023-10-17T02:03:01.580612+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"108'508520","reported_seq":1016273,"reported_epoch":108,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:03:01.679555+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:03:01.679555+0000","last_peered":"2023-10-17T02:03:01.679555+0000","last_clean":"2023-10-17T02:03:01.679555+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:03:01.679555+0000","last_undegraded":"2023-10-17T02:03:01.679555+0000","last_fullsized":"2023-10-17T02:03:01.679555+0000","mapping_epoch":16,"log_start":"106'498516","ondisk_log_start":"106'498516","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":304,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 96s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6962630675,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":508550,"num_read_kb":2002944,"num_write":507080,"num_write_kb":2028314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"108'10142","reported_seq":14253,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-17T02:03:01.506945+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:03:01.506945+0000","last_peered":"2023-10-17T02:03:01.506945+0000","last_clean":"2023-10-17T02:03:01.506945+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:03:01.506945+0000","last_undegraded":"2023-10-17T02:03:01.506945+0000","last_fullsized":"2023-10-17T02:03:01.506945+0000","mapping_epoch":13,"log_start":"40'100","ondisk_log_start":"40'100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":285,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-10-17T02:02:41.509656+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:02:41.509656+0000","last_peered":"2023-10-17T02:02:41.509656+0000","last_clean":"2023-10-17T02:02:41.509656+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:02:41.509656+0000","last_undegraded":"2023-10-17T02:02:41.509656+0000","last_fullsized":"2023-10-17T02:02:41.509656+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28006359059,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2042724,"num_read_kb":8047200,"num_write":2040775,"num_write_kb":8163094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16721625088,"data_stored":20122484774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40131,"ondisk_log_size":40131,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":47232,"ondisk_log_size":47232,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739343,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9873620,"kb_used_data":8167116,"kb_used_omap":764,"kb_used_meta":1705731,"kb_avail":84498220,"statfs":{"total":96636764160,"available":86526177280,"internally_reserved":0,"allocated":8363126784,"data_stored":10062937896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":782639,"internal_metadata":1746669265},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,4,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:02:23 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.439,"15min":0.41699999999999998},"min":{"1min":0.18099999999999999,"5min":0.17799999999999999,"15min":0.16900000000000001},"max":{"1min":8.4190000000000005,"5min":8.4190000000000005,"15min":8.4190000000000005},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.749,"5min":0.497,"15min":0.5},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.188},"max":{"1min":8.3559999999999999,"5min":8.3559999999999999,"15min":8.3559999999999999},"last":0.25800000000000001}]},{"osd":1,"last update":"Tue Oct 17 02:02:55 2023","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.435,"15min":0.47499999999999998},"min":{"1min":0.22800000000000001,"5min":0.218,"15min":0.20300000000000001},"max":{"1min":0.54300000000000004,"5min":5.2169999999999996,"15min":9.5429999999999993},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.57899999999999996,"15min":0.57499999999999996},"min":{"1min":0.23799999999999999,"5min":0.23200000000000001,"15min":0.19500000000000001},"max":{"1min":8.3930000000000007,"5min":8.3930000000000007,"15min":9.5999999999999996},"last":0.55800000000000005}]}]},{"osd":1,"up_from":8,"seq":34359739339,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5562108,"kb_used_data":4019316,"kb_used_omap":362,"kb_used_meta":1542421,"kb_avail":88809732,"statfs":{"total":96636764160,"available":90941165568,"internally_reserved":0,"allocated":4115779584,"data_stored":4953364295,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":371063,"internal_metadata":1579439753},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,1,3,2,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:02:13 2023","interfaces":[{"interface":"back","average":{"1min":0.28399999999999997,"5min":0.33900000000000002,"15min":0.52600000000000002},"min":{"1min":0.193,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.42699999999999999,"5min":2.4809999999999999,"15min":23.177},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.38800000000000001,"15min":0.57799999999999996},"min":{"1min":0.249,"5min":0.22,"15min":0.22},"max":{"1min":0.46800000000000003,"5min":2.512,"15min":23.207999999999998},"last":0.30599999999999999}]},{"osd":2,"last update":"Tue Oct 17 02:02:13 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53600000000000003,"15min":0.50700000000000001},"min":{"1min":0.217,"5min":0.19700000000000001,"15min":0.188},"max":{"1min":4.4429999999999996,"5min":6.8230000000000004,"15min":6.8230000000000004},"last":0.435},{"interface":"front","average":{"1min":0.30399999999999999,"5min":0.441,"15min":0.48099999999999998},"min":{"1min":0.22600000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.38,"5min":6.7800000000000002,"15min":6.7800000000000002},"last":0.39400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739339,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6015060,"kb_used_data":4145164,"kb_used_omap":416,"kb_used_meta":1869471,"kb_avail":88356780,"statfs":{"total":96636764160,"available":90477342720,"internally_reserved":0,"allocated":4244647936,"data_stored":5107078158,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":426086,"internal_metadata":1914339226},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,10,4,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:02:52 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.45700000000000002,"15min":0.44600000000000001},"min":{"1min":0.182,"5min":0.17000000000000001,"15min":0.16200000000000001},"max":{"1min":0.52900000000000003,"5min":4.101,"15min":5.9829999999999997},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.312,"5min":0.35999999999999999,"15min":0.48199999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.191},"max":{"1min":0.52400000000000002,"5min":2.2970000000000002,"15min":6.0750000000000002},"last":0.22}]},{"osd":2,"last update":"Tue Oct 17 02:02:58 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.51800000000000002,"15min":0.41799999999999998},"min":{"1min":0.252,"5min":0.20699999999999999,"15min":0.191},"max":{"1min":2.5,"5min":6.0010000000000003,"15min":6.0010000000000003},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.54900000000000004,"15min":0.59699999999999998},"min":{"1min":0.249,"5min":0.23100000000000001,"15min":0.19400000000000001},"max":{"1min":3.903,"5min":8.0169999999999995,"15min":10.378},"last":0.26000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4243800064,"data_stored":5106397184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4115230720,"data_stored":4953063443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8362594304,"data_stored":10063024147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:03:05.956 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:03:05.957 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:03:05.957 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:03:25.958 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:03:26.447 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:03:26.448 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:03:26.479 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2545,"stamp":"2023-10-17T02:03:25.509801+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28036705112,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2073881,"num_read_kb":8096501,"num_write":2048738,"num_write_kb":8194603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87530,"ondisk_log_size":87530,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21600372,"kb_used_data":16390236,"kb_used_omap":1542,"kb_used_meta":5208569,"kb_avail":261515148,"statfs":{"total":289910292480,"available":267791511552,"internally_reserved":0,"allocated":16783601664,"data_stored":20183427709,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1579788,"internal_metadata":5333574900},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,3,22,16,28],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15282176,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3575,"num_read_kb":14224,"num_write":3761,"num_write_kb":15044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001932"},"pg_stats":[{"pgid":"2.7","version":"108'13477","reported_seq":19041,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-17T02:03:21.517740+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:03:21.517740+0000","last_peered":"2023-10-17T02:03:21.517740+0000","last_clean":"2023-10-17T02:03:21.517740+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:03:21.517740+0000","last_undegraded":"2023-10-17T02:03:21.517740+0000","last_fullsized":"2023-10-17T02:03:21.517740+0000","mapping_epoch":13,"log_start":"73'3400","ondisk_log_start":"73'3400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"108'10153","reported_seq":15500,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-17T02:03:21.513687+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:03:21.513687+0000","last_peered":"2023-10-17T02:03:21.513687+0000","last_clean":"2023-10-17T02:03:21.513687+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:03:21.513687+0000","last_undegraded":"2023-10-17T02:03:21.513687+0000","last_fullsized":"2023-10-17T02:03:21.513687+0000","mapping_epoch":13,"log_start":"40'100","ondisk_log_start":"40'100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"108'6806","reported_seq":9677,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-17T02:03:21.505926+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:03:21.505926+0000","last_peered":"2023-10-17T02:03:21.505926+0000","last_clean":"2023-10-17T02:03:21.505926+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:03:21.505926+0000","last_undegraded":"2023-10-17T02:03:21.505926+0000","last_fullsized":"2023-10-17T02:03:21.505926+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6806,"log_dups_size":0,"ondisk_log_size":6806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"108'10186","reported_seq":14941,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-17T02:03:21.522210+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:03:21.522210+0000","last_peered":"2023-10-17T02:03:21.522210+0000","last_clean":"2023-10-17T02:03:21.522210+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:03:21.522210+0000","last_undegraded":"2023-10-17T02:03:21.522210+0000","last_fullsized":"2023-10-17T02:03:21.522210+0000","mapping_epoch":13,"log_start":"40'100","ondisk_log_start":"40'100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"108'521955","reported_seq":1040154,"reported_epoch":108,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:03:21.656014+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T02:03:21.656014+0000","last_peered":"2023-10-17T02:03:21.656014+0000","last_clean":"2023-10-17T02:03:21.656014+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:03:21.656014+0000","last_undegraded":"2023-10-17T02:03:21.656014+0000","last_fullsized":"2023-10-17T02:03:21.656014+0000","mapping_epoch":16,"log_start":"106'511910","ondisk_log_start":"106'511910","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2328,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 908s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7107739648,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":518935,"num_read_kb":2043612,"num_write":519342,"num_write_kb":2077368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":264,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-10-17T02:01:57.319523+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:01:57.319523+0000","last_peered":"2023-10-17T02:01:57.319523+0000","last_clean":"2023-10-17T02:01:57.319523+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:01:57.319523+0000","last_undegraded":"2023-10-17T02:01:57.319523+0000","last_fullsized":"2023-10-17T02:01:57.319523+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"108'502046","reported_seq":1001537,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-17T02:03:21.764653+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:03:21.764653+0000","last_peered":"2023-10-17T02:03:21.764653+0000","last_clean":"2023-10-17T02:03:21.764653+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:03:21.764653+0000","last_undegraded":"2023-10-17T02:03:21.764653+0000","last_fullsized":"2023-10-17T02:03:21.764653+0000","mapping_epoch":16,"log_start":"106'492019","ondisk_log_start":"106'492019","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6833516544,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":500178,"num_read_kb":1971664,"num_write":499396,"num_write_kb":1997584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2835,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-17T02:03:21.983828+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:03:21.983828+0000","last_peered":"2023-10-17T02:03:21.983828+0000","last_clean":"2023-10-17T02:03:21.983828+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:03:21.983828+0000","last_undegraded":"2023-10-17T02:03:21.983828+0000","last_fullsized":"2023-10-17T02:03:21.983828+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"108'523238","reported_seq":1042946,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-17T02:03:21.606911+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:03:21.606911+0000","last_peered":"2023-10-17T02:03:21.606911+0000","last_clean":"2023-10-17T02:03:21.606911+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:03:21.606911+0000","last_undegraded":"2023-10-17T02:03:21.606911+0000","last_fullsized":"2023-10-17T02:03:21.606911+0000","mapping_epoch":16,"log_start":"106'513225","ondisk_log_start":"106'513225","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7124897792,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":520434,"num_read_kb":2050364,"num_write":520484,"num_write_kb":2081936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4252,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-17T02:03:21.986174+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:03:21.986174+0000","last_peered":"2023-10-17T02:03:21.986174+0000","last_clean":"2023-10-17T02:03:21.986174+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:03:21.986174+0000","last_undegraded":"2023-10-17T02:03:21.986174+0000","last_fullsized":"2023-10-17T02:03:21.986174+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"108'510355","reported_seq":1019941,"reported_epoch":108,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:03:21.801895+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:03:21.801895+0000","last_peered":"2023-10-17T02:03:21.801895+0000","last_clean":"2023-10-17T02:03:21.801895+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:03:21.801895+0000","last_undegraded":"2023-10-17T02:03:21.801895+0000","last_fullsized":"2023-10-17T02:03:21.801895+0000","mapping_epoch":16,"log_start":"106'500316","ondisk_log_start":"106'500316","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":354,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 116s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6969958419,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":510372,"num_read_kb":2010208,"num_write":508886,"num_write_kb":2035538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"108'10157","reported_seq":14277,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-17T02:03:21.540683+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:03:21.540683+0000","last_peered":"2023-10-17T02:03:21.540683+0000","last_clean":"2023-10-17T02:03:21.540683+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:03:21.540683+0000","last_undegraded":"2023-10-17T02:03:21.540683+0000","last_fullsized":"2023-10-17T02:03:21.540683+0000","mapping_epoch":13,"log_start":"40'100","ondisk_log_start":"40'100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":285,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-10-17T02:02:41.509656+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:02:41.509656+0000","last_peered":"2023-10-17T02:02:41.509656+0000","last_clean":"2023-10-17T02:02:41.509656+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:02:41.509656+0000","last_undegraded":"2023-10-17T02:02:41.509656+0000","last_fullsized":"2023-10-17T02:02:41.509656+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28036112403,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2049919,"num_read_kb":8075848,"num_write":2048108,"num_write_kb":8192426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16781701120,"data_stored":20182560806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40124,"ondisk_log_size":40124,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":47307,"ondisk_log_size":47307,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739347,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9935492,"kb_used_data":8196476,"kb_used_omap":764,"kb_used_meta":1738243,"kb_avail":84436348,"statfs":{"total":96636764160,"available":86462820352,"internally_reserved":0,"allocated":8393191424,"data_stored":10093002536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":782639,"internal_metadata":1779961553},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,6,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:02:23 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.439,"15min":0.41699999999999998},"min":{"1min":0.18099999999999999,"5min":0.17799999999999999,"15min":0.16900000000000001},"max":{"1min":8.4190000000000005,"5min":8.4190000000000005,"15min":8.4190000000000005},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.749,"5min":0.497,"15min":0.5},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.188},"max":{"1min":8.3559999999999999,"5min":8.3559999999999999,"15min":8.3559999999999999},"last":0.38400000000000001}]},{"osd":1,"last update":"Tue Oct 17 02:02:55 2023","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.435,"15min":0.47499999999999998},"min":{"1min":0.22800000000000001,"5min":0.218,"15min":0.20300000000000001},"max":{"1min":0.54300000000000004,"5min":5.2169999999999996,"15min":9.5429999999999993},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.57899999999999996,"15min":0.57499999999999996},"min":{"1min":0.23799999999999999,"5min":0.23200000000000001,"15min":0.19500000000000001},"max":{"1min":8.3930000000000007,"5min":8.3930000000000007,"15min":9.5999999999999996},"last":0.317}]}]},{"osd":1,"up_from":8,"seq":34359739343,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5598864,"kb_used_data":4033736,"kb_used_omap":362,"kb_used_meta":1564757,"kb_avail":88772976,"statfs":{"total":96636764160,"available":90903527424,"internally_reserved":0,"allocated":4130545664,"data_stored":4968130375,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":371063,"internal_metadata":1602311817},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,11,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:03:16 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.43099999999999999,"15min":0.46899999999999997},"min":{"1min":0.20399999999999999,"5min":0.16600000000000001,"15min":0.16300000000000001},"max":{"1min":5.5869999999999997,"5min":5.5869999999999997,"15min":6.6890000000000001},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.48099999999999998,"15min":0.52100000000000002},"min":{"1min":0.27300000000000002,"5min":0.23000000000000001,"15min":0.22},"max":{"1min":5.6360000000000001,"5min":5.6360000000000001,"15min":6.7539999999999996},"last":0.318}]},{"osd":2,"last update":"Tue Oct 17 02:03:16 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.55700000000000005,"15min":0.51000000000000001},"min":{"1min":0.223,"5min":0.216,"15min":0.188},"max":{"1min":2.653,"5min":6.8230000000000004,"15min":6.8230000000000004},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.49299999999999999,"15min":0.48399999999999999},"min":{"1min":0.23499999999999999,"5min":0.22600000000000001,"15min":0.20100000000000001},"max":{"1min":2.9870000000000001,"5min":6.7800000000000002,"15min":6.7800000000000002},"last":0.27000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739343,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6066016,"kb_used_data":4160024,"kb_used_omap":416,"kb_used_meta":1905567,"kb_avail":88305824,"statfs":{"total":96636764160,"available":90425163776,"internally_reserved":0,"allocated":4259864576,"data_stored":5122294798,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":426086,"internal_metadata":1951301530},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,5,2,25],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:02:52 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.45700000000000002,"15min":0.44600000000000001},"min":{"1min":0.182,"5min":0.17000000000000001,"15min":0.16200000000000001},"max":{"1min":0.52900000000000003,"5min":4.101,"15min":5.9829999999999997},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.312,"5min":0.35999999999999999,"15min":0.48199999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.191},"max":{"1min":0.52400000000000002,"5min":2.2970000000000002,"15min":6.0750000000000002},"last":0.55600000000000005}]},{"osd":2,"last update":"Tue Oct 17 02:02:58 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.51800000000000002,"15min":0.41799999999999998},"min":{"1min":0.252,"5min":0.20699999999999999,"15min":0.191},"max":{"1min":2.5,"5min":6.0010000000000003,"15min":6.0010000000000003},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.54900000000000004,"15min":0.59699999999999998},"min":{"1min":0.249,"5min":0.23100000000000001,"15min":0.19400000000000001},"max":{"1min":3.903,"5min":8.0169999999999995,"15min":10.378},"last":0.59199999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4259127296,"data_stored":5121724416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4129931264,"data_stored":4967763987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8392642560,"data_stored":10093072403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:03:26.481 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:03:26.481 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:03:26.482 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:03:37.876 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:03:37.873+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T02:03:37.903 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:03:37.881+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T02:03:37.908 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:03:37.905+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T02:03:37.909 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:03:37.905+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T02:03:37.909 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:03:37.905+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T02:03:37.909 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:03:37.905+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T02:03:37.909 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:03:37.905+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T02:03:37.910 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:03:37.905+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T02:03:37.911 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:03:37.905+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T02:03:37.912 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:03:37.905+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T02:03:37.912 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:03:37.905+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T02:03:37.912 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:03:37.905+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T02:03:37.912 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:03:37.905+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T02:03:37.913 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:03:37.905+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T02:03:46.483 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:03:47.042 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:03:47.042 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:03:47.082 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2555,"stamp":"2023-10-17T02:03:45.517827+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28066523992,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2081030,"num_read_kb":8124893,"num_write":2056084,"num_write_kb":8223987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87359,"ondisk_log_size":87359,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21840644,"kb_used_data":16448876,"kb_used_omap":1542,"kb_used_meta":5390201,"kb_avail":261274876,"statfs":{"total":289910292480,"available":267545473024,"internally_reserved":0,"allocated":16843649024,"data_stored":20243475069,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1579788,"internal_metadata":5519566068},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,3,3,3,6,22,14],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14589952,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3487,"num_read_kb":13840,"num_write":3598,"num_write_kb":14392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002029"},"pg_stats":[{"pgid":"2.7","version":"108'13541","reported_seq":19121,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-17T02:03:40.426768+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:03:40.426768+0000","last_peered":"2023-10-17T02:03:40.426768+0000","last_clean":"2023-10-17T02:03:40.426768+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:03:40.426768+0000","last_undegraded":"2023-10-17T02:03:40.426768+0000","last_fullsized":"2023-10-17T02:03:40.426768+0000","mapping_epoch":13,"log_start":"73'3500","ondisk_log_start":"73'3500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"108'10201","reported_seq":15560,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-17T02:03:40.414734+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:03:40.414734+0000","last_peered":"2023-10-17T02:03:40.414734+0000","last_clean":"2023-10-17T02:03:40.414734+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:03:40.414734+0000","last_undegraded":"2023-10-17T02:03:40.414734+0000","last_fullsized":"2023-10-17T02:03:40.414734+0000","mapping_epoch":13,"log_start":"42'200","ondisk_log_start":"42'200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"108'6838","reported_seq":9717,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-17T02:03:40.406392+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:03:40.406392+0000","last_peered":"2023-10-17T02:03:40.406392+0000","last_clean":"2023-10-17T02:03:40.406392+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:03:40.406392+0000","last_undegraded":"2023-10-17T02:03:40.406392+0000","last_fullsized":"2023-10-17T02:03:40.406392+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6838,"log_dups_size":0,"ondisk_log_size":6838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"108'10234","reported_seq":15001,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-17T02:03:40.408890+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:03:40.408890+0000","last_peered":"2023-10-17T02:03:40.408890+0000","last_clean":"2023-10-17T02:03:40.408890+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:03:40.408890+0000","last_undegraded":"2023-10-17T02:03:40.408890+0000","last_fullsized":"2023-10-17T02:03:40.408890+0000","mapping_epoch":13,"log_start":"42'200","ondisk_log_start":"42'200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"108'523876","reported_seq":1043883,"reported_epoch":108,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:03:41.752344+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T02:03:41.752344+0000","last_peered":"2023-10-17T02:03:41.752344+0000","last_clean":"2023-10-17T02:03:41.752344+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:03:41.752344+0000","last_undegraded":"2023-10-17T02:03:41.752344+0000","last_fullsized":"2023-10-17T02:03:41.752344+0000","mapping_epoch":16,"log_start":"106'513810","ondisk_log_start":"106'513810","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2366,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 928s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7115472896,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":520738,"num_read_kb":2050768,"num_write":521243,"num_write_kb":2084972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":264,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-10-17T02:01:57.319523+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:01:57.319523+0000","last_peered":"2023-10-17T02:01:57.319523+0000","last_clean":"2023-10-17T02:01:57.319523+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:01:57.319523+0000","last_undegraded":"2023-10-17T02:01:57.319523+0000","last_fullsized":"2023-10-17T02:01:57.319523+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"108'503839","reported_seq":1005148,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-17T02:03:41.889269+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:03:41.889269+0000","last_peered":"2023-10-17T02:03:41.889269+0000","last_clean":"2023-10-17T02:03:41.889269+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:03:41.889269+0000","last_undegraded":"2023-10-17T02:03:41.889269+0000","last_fullsized":"2023-10-17T02:03:41.889269+0000","mapping_epoch":16,"log_start":"106'493819","ondisk_log_start":"106'493819","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6840774656,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":501994,"num_read_kb":1978884,"num_write":501189,"num_write_kb":2004756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2843,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-17T02:03:42.466705+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:03:42.466705+0000","last_peered":"2023-10-17T02:03:42.466705+0000","last_clean":"2023-10-17T02:03:42.466705+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:03:42.466705+0000","last_undegraded":"2023-10-17T02:03:42.466705+0000","last_fullsized":"2023-10-17T02:03:42.466705+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"108'525050","reported_seq":1046510,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-17T02:03:41.751775+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:03:41.751775+0000","last_peered":"2023-10-17T02:03:41.751775+0000","last_clean":"2023-10-17T02:03:41.751775+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:03:41.751775+0000","last_undegraded":"2023-10-17T02:03:41.751775+0000","last_fullsized":"2023-10-17T02:03:41.751775+0000","mapping_epoch":16,"log_start":"106'515025","ondisk_log_start":"106'515025","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7132205056,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":522181,"num_read_kb":2057300,"num_write":522296,"num_write_kb":2089184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4264,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-17T02:03:42.464124+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:03:42.464124+0000","last_peered":"2023-10-17T02:03:42.464124+0000","last_clean":"2023-10-17T02:03:42.464124+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:03:42.464124+0000","last_undegraded":"2023-10-17T02:03:42.464124+0000","last_fullsized":"2023-10-17T02:03:42.464124+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"108'512218","reported_seq":1023594,"reported_epoch":108,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:03:41.903673+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:03:41.903673+0000","last_peered":"2023-10-17T02:03:41.903673+0000","last_clean":"2023-10-17T02:03:41.903673+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:03:41.903673+0000","last_undegraded":"2023-10-17T02:03:41.903673+0000","last_fullsized":"2023-10-17T02:03:41.903673+0000","mapping_epoch":16,"log_start":"106'502216","ondisk_log_start":"106'502216","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":400,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 136s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6977478675,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":512155,"num_read_kb":2017288,"num_write":510726,"num_write_kb":2042898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"108'10205","reported_seq":14337,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-17T02:03:40.525841+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:03:40.525841+0000","last_peered":"2023-10-17T02:03:40.525841+0000","last_clean":"2023-10-17T02:03:40.525841+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:03:40.525841+0000","last_undegraded":"2023-10-17T02:03:40.525841+0000","last_fullsized":"2023-10-17T02:03:40.525841+0000","mapping_epoch":13,"log_start":"42'200","ondisk_log_start":"42'200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":285,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-10-17T02:02:41.509656+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:02:41.509656+0000","last_peered":"2023-10-17T02:02:41.509656+0000","last_clean":"2023-10-17T02:02:41.509656+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:02:41.509656+0000","last_undegraded":"2023-10-17T02:02:41.509656+0000","last_fullsized":"2023-10-17T02:02:41.509656+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28065931283,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2057068,"num_read_kb":8104240,"num_write":2055454,"num_write_kb":8221810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16841867264,"data_stored":20242726950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40113,"ondisk_log_size":40113,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":47147,"ondisk_log_size":47147,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739351,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10043932,"kb_used_data":8225812,"kb_used_omap":764,"kb_used_meta":1817347,"kb_avail":84327908,"statfs":{"total":96636764160,"available":86351777792,"internally_reserved":0,"allocated":8423231488,"data_stored":10123042600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":782639,"internal_metadata":1860964049},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:03:25 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.41999999999999998,"15min":0.41299999999999998},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":1.3660000000000001,"5min":8.4190000000000005,"15min":8.4190000000000005},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51200000000000001,"15min":0.49299999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.188},"max":{"1min":3.629,"5min":8.3559999999999999,"15min":8.3559999999999999},"last":0.39700000000000002}]},{"osd":1,"last update":"Tue Oct 17 02:02:55 2023","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.435,"15min":0.47499999999999998},"min":{"1min":0.22800000000000001,"5min":0.218,"15min":0.20300000000000001},"max":{"1min":0.54300000000000004,"5min":5.2169999999999996,"15min":9.5429999999999993},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.57899999999999996,"15min":0.57499999999999996},"min":{"1min":0.23799999999999999,"5min":0.23200000000000001,"15min":0.19500000000000001},"max":{"1min":8.3930000000000007,"5min":8.3930000000000007,"15min":9.5999999999999996},"last":0.33100000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739347,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5663324,"kb_used_data":4048212,"kb_used_omap":362,"kb_used_meta":1614741,"kb_avail":88708516,"statfs":{"total":96636764160,"available":90837520384,"internally_reserved":0,"allocated":4145369088,"data_stored":4982953799,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":371063,"internal_metadata":1653495433},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,0,3,15,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:03:16 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.43099999999999999,"15min":0.46899999999999997},"min":{"1min":0.20399999999999999,"5min":0.16600000000000001,"15min":0.16300000000000001},"max":{"1min":5.5869999999999997,"5min":5.5869999999999997,"15min":6.6890000000000001},"last":0.316},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.48099999999999998,"15min":0.52100000000000002},"min":{"1min":0.27300000000000002,"5min":0.23000000000000001,"15min":0.22},"max":{"1min":5.6360000000000001,"5min":5.6360000000000001,"15min":6.7539999999999996},"last":0.377}]},{"osd":2,"last update":"Tue Oct 17 02:03:16 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.55700000000000005,"15min":0.51000000000000001},"min":{"1min":0.223,"5min":0.216,"15min":0.188},"max":{"1min":2.653,"5min":6.8230000000000004,"15min":6.8230000000000004},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.49299999999999999,"15min":0.48399999999999999},"min":{"1min":0.23499999999999999,"5min":0.22600000000000001,"15min":0.20100000000000001},"max":{"1min":2.9870000000000001,"5min":6.7800000000000002,"15min":6.7800000000000002},"last":0.27500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739347,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6133388,"kb_used_data":4174852,"kb_used_omap":416,"kb_used_meta":1958111,"kb_avail":88238452,"statfs":{"total":96636764160,"available":90356174848,"internally_reserved":0,"allocated":4275048448,"data_stored":5137478670,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":426086,"internal_metadata":2005106586},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,3,2,7,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:02:52 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.45700000000000002,"15min":0.44600000000000001},"min":{"1min":0.182,"5min":0.17000000000000001,"15min":0.16200000000000001},"max":{"1min":0.52900000000000003,"5min":4.101,"15min":5.9829999999999997},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.312,"5min":0.35999999999999999,"15min":0.48199999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.191},"max":{"1min":0.52400000000000002,"5min":2.2970000000000002,"15min":6.0750000000000002},"last":0.30099999999999999}]},{"osd":2,"last update":"Tue Oct 17 02:02:58 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.51800000000000002,"15min":0.41799999999999998},"min":{"1min":0.252,"5min":0.20699999999999999,"15min":0.191},"max":{"1min":2.5,"5min":6.0010000000000003,"15min":6.0010000000000003},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.54900000000000004,"15min":0.59699999999999998},"min":{"1min":0.249,"5min":0.23100000000000001,"15min":0.19400000000000001},"max":{"1min":3.903,"5min":8.0169999999999995,"15min":10.378},"last":0.26900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4274339840,"data_stored":5136936960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4144807936,"data_stored":4982640659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8422719488,"data_stored":10123149331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:03:47.084 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:03:47.085 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:03:47.085 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:04:07.086 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:04:07.659 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:04:07.660 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:04:07.687 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2567,"stamp":"2023-10-17T02:04:07.521470+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28103854936,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2090356,"num_read_kb":8161973,"num_write":2065268,"num_write_kb":8260723,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87588,"ondisk_log_size":87588,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22212168,"kb_used_data":16514988,"kb_used_omap":1542,"kb_used_meta":5695609,"kb_avail":260903352,"statfs":{"total":289910292480,"available":267165032448,"internally_reserved":0,"allocated":16911347712,"data_stored":20311160554,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1579788,"internal_metadata":5832303860},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,3,7,14,20,11,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14422016,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3727,"num_read_kb":14852,"num_write":3555,"num_write_kb":14220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001587"},"pg_stats":[{"pgid":"2.7","version":"109'13613","reported_seq":19214,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:04:05.914719+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:04:05.914719+0000","last_peered":"2023-10-17T02:04:05.914719+0000","last_clean":"2023-10-17T02:04:05.914719+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:04:05.914719+0000","last_undegraded":"2023-10-17T02:04:05.914719+0000","last_fullsized":"2023-10-17T02:04:05.914719+0000","mapping_epoch":13,"log_start":"75'3600","ondisk_log_start":"75'3600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"109'10255","reported_seq":15630,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:04:05.909558+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:04:05.909558+0000","last_peered":"2023-10-17T02:04:05.909558+0000","last_clean":"2023-10-17T02:04:05.909558+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:04:05.909558+0000","last_undegraded":"2023-10-17T02:04:05.909558+0000","last_fullsized":"2023-10-17T02:04:05.909558+0000","mapping_epoch":13,"log_start":"42'200","ondisk_log_start":"42'200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"109'6874","reported_seq":9765,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:04:05.896156+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:04:05.896156+0000","last_peered":"2023-10-17T02:04:05.896156+0000","last_clean":"2023-10-17T02:04:05.896156+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:04:05.896156+0000","last_undegraded":"2023-10-17T02:04:05.896156+0000","last_fullsized":"2023-10-17T02:04:05.896156+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6874,"log_dups_size":0,"ondisk_log_size":6874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"109'10288","reported_seq":15072,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:04:05.907845+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:04:05.907845+0000","last_peered":"2023-10-17T02:04:05.907845+0000","last_clean":"2023-10-17T02:04:05.907845+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:04:05.907845+0000","last_undegraded":"2023-10-17T02:04:05.907845+0000","last_fullsized":"2023-10-17T02:04:05.907845+0000","mapping_epoch":13,"log_start":"42'200","ondisk_log_start":"42'200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"109'526222","reported_seq":1048606,"reported_epoch":109,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:04:06.907444+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T02:04:06.907444+0000","last_peered":"2023-10-17T02:04:06.907444+0000","last_clean":"2023-10-17T02:04:06.907444+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:04:06.907444+0000","last_undegraded":"2023-10-17T02:04:06.907444+0000","last_fullsized":"2023-10-17T02:04:06.907444+0000","mapping_epoch":16,"log_start":"107'516210","ondisk_log_start":"107'516210","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2428,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 953s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7124877312,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":523105,"num_read_kb":2060196,"num_write":523553,"num_write_kb":2094212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":266,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-17T02:03:56.846257+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:03:56.846257+0000","last_peered":"2023-10-17T02:03:56.846257+0000","last_clean":"2023-10-17T02:03:56.846257+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:03:56.846257+0000","last_undegraded":"2023-10-17T02:03:56.846257+0000","last_fullsized":"2023-10-17T02:03:56.846257+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"109'506073","reported_seq":1009624,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:04:07.094877+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:04:07.094877+0000","last_peered":"2023-10-17T02:04:07.094877+0000","last_clean":"2023-10-17T02:04:07.094877+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:04:07.094877+0000","last_undegraded":"2023-10-17T02:04:07.094877+0000","last_fullsized":"2023-10-17T02:04:07.094877+0000","mapping_epoch":16,"log_start":"107'496019","ondisk_log_start":"107'496019","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6849859584,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":504241,"num_read_kb":1987808,"num_write":503423,"num_write_kb":2013692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2853,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:04:02.998871+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:04:02.998871+0000","last_peered":"2023-10-17T02:04:02.998871+0000","last_clean":"2023-10-17T02:04:02.998871+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:04:02.998871+0000","last_undegraded":"2023-10-17T02:04:02.998871+0000","last_fullsized":"2023-10-17T02:04:02.998871+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"109'527381","reported_seq":1051254,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:04:06.904135+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:04:06.904135+0000","last_peered":"2023-10-17T02:04:06.904135+0000","last_clean":"2023-10-17T02:04:06.904135+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:04:06.904135+0000","last_undegraded":"2023-10-17T02:04:06.904135+0000","last_fullsized":"2023-10-17T02:04:06.904135+0000","mapping_epoch":16,"log_start":"107'517325","ondisk_log_start":"107'517325","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7141675008,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":524595,"num_read_kb":2066876,"num_write":524627,"num_write_kb":2098508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4277,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:04:02.948685+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:04:02.948685+0000","last_peered":"2023-10-17T02:04:02.948685+0000","last_clean":"2023-10-17T02:04:02.948685+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:04:02.948685+0000","last_undegraded":"2023-10-17T02:04:02.948685+0000","last_fullsized":"2023-10-17T02:04:02.948685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"109'514566","reported_seq":1028245,"reported_epoch":109,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:04:07.060859+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:04:07.060859+0000","last_peered":"2023-10-17T02:04:07.060859+0000","last_clean":"2023-10-17T02:04:07.060859+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:04:07.060859+0000","last_undegraded":"2023-10-17T02:04:07.060859+0000","last_fullsized":"2023-10-17T02:04:07.060859+0000","mapping_epoch":16,"log_start":"107'504516","ondisk_log_start":"107'504516","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":474,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 161s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6986850323,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":514453,"num_read_kb":2026440,"num_write":513035,"num_write_kb":2052134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"109'10259","reported_seq":14408,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:04:05.933607+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:04:05.933607+0000","last_peered":"2023-10-17T02:04:05.933607+0000","last_clean":"2023-10-17T02:04:05.933607+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:04:05.933607+0000","last_undegraded":"2023-10-17T02:04:05.933607+0000","last_fullsized":"2023-10-17T02:04:05.933607+0000","mapping_epoch":13,"log_start":"42'200","ondisk_log_start":"42'200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":286,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-17T02:02:52.335311+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:02:52.335311+0000","last_peered":"2023-10-17T02:02:52.335311+0000","last_clean":"2023-10-17T02:02:52.335311+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:02:52.335311+0000","last_undegraded":"2023-10-17T02:02:52.335311+0000","last_fullsized":"2023-10-17T02:02:52.335311+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28103262227,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2066394,"num_read_kb":8141320,"num_write":2064638,"num_write_kb":8258546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16909787136,"data_stored":20310646822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40172,"ondisk_log_size":40172,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":47317,"ondisk_log_size":47317,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739355,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10173284,"kb_used_data":8255192,"kb_used_omap":764,"kb_used_meta":1917315,"kb_avail":84198556,"statfs":{"total":96636764160,"available":86219321344,"internally_reserved":0,"allocated":8453316608,"data_stored":10153123319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":782639,"internal_metadata":1963331281},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,4,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:03:25 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.41999999999999998,"15min":0.41299999999999998},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":1.3660000000000001,"5min":8.4190000000000005,"15min":8.4190000000000005},"last":0.214},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51200000000000001,"15min":0.49299999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.188},"max":{"1min":3.629,"5min":8.3559999999999999,"15min":8.3559999999999999},"last":0.25}]},{"osd":1,"last update":"Tue Oct 17 02:03:59 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.41299999999999998,"15min":0.48099999999999998},"min":{"1min":0.23999999999999999,"5min":0.218,"15min":0.20699999999999999},"max":{"1min":6.202,"5min":6.202,"15min":9.5429999999999993},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.54100000000000004,"15min":0.57599999999999996},"min":{"1min":0.26100000000000001,"5min":0.23200000000000001,"15min":0.19500000000000001},"max":{"1min":6.1699999999999999,"5min":8.3930000000000007,"15min":9.5999999999999996},"last":0.52300000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739352,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5739804,"kb_used_data":4066388,"kb_used_omap":362,"kb_used_meta":1673045,"kb_avail":88632036,"statfs":{"total":96636764160,"available":90759204864,"internally_reserved":0,"allocated":4163981312,"data_stored":5001561622,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":371063,"internal_metadata":1713198729},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,6,8,7,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:03:16 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.43099999999999999,"15min":0.46899999999999997},"min":{"1min":0.20399999999999999,"5min":0.16600000000000001,"15min":0.16300000000000001},"max":{"1min":5.5869999999999997,"5min":5.5869999999999997,"15min":6.6890000000000001},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.48099999999999998,"15min":0.52100000000000002},"min":{"1min":0.27300000000000002,"5min":0.23000000000000001,"15min":0.22},"max":{"1min":5.6360000000000001,"5min":5.6360000000000001,"15min":6.7539999999999996},"last":0.314}]},{"osd":2,"last update":"Tue Oct 17 02:03:16 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.55700000000000005,"15min":0.51000000000000001},"min":{"1min":0.223,"5min":0.216,"15min":0.188},"max":{"1min":2.653,"5min":6.8230000000000004,"15min":6.8230000000000004},"last":0.252},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.49299999999999999,"15min":0.48399999999999999},"min":{"1min":0.23499999999999999,"5min":0.22600000000000001,"15min":0.20100000000000001},"max":{"1min":2.9870000000000001,"5min":6.7800000000000002,"15min":6.7800000000000002},"last":0.28000000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739352,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6299080,"kb_used_data":4193408,"kb_used_omap":416,"kb_used_meta":2105247,"kb_avail":88072760,"statfs":{"total":96636764160,"available":90186506240,"internally_reserved":0,"allocated":4294049792,"data_stored":5156475613,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":426086,"internal_metadata":2155773850},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,3,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:03:52 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.46000000000000002,"15min":0.42199999999999999},"min":{"1min":0.19,"5min":0.17999999999999999,"15min":0.17000000000000001},"max":{"1min":1.675,"5min":4.101,"15min":5.9829999999999997},"last":0.255},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.40500000000000003,"15min":0.48199999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":6.3460000000000001,"5min":6.3460000000000001,"15min":6.3460000000000001},"last":0.31900000000000001}]},{"osd":2,"last update":"Tue Oct 17 02:04:01 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.53100000000000003,"15min":0.42599999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":1.3340000000000001,"5min":6.0010000000000003,"15min":6.0010000000000003},"last":4.3460000000000001},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.60699999999999998,"15min":0.61199999999999999},"min":{"1min":0.23699999999999999,"5min":0.23100000000000001,"15min":0.19400000000000001},"max":{"1min":6.3899999999999997,"5min":8.0169999999999995,"15min":10.378},"last":0.28499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4293349376,"data_stored":5155946496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4163416064,"data_stored":5001248787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8453021696,"data_stored":10153451539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:04:07.689 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:04:07.689 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:04:07.689 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:04:27.690 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:04:28.231 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:04:28.232 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:04:28.256 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2577,"stamp":"2023-10-17T02:04:27.524733+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28133546840,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2097710,"num_read_kb":8191269,"num_write":2072580,"num_write_kb":8289971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87593,"ondisk_log_size":87593,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21628492,"kb_used_data":16572852,"kb_used_omap":1542,"kb_used_meta":5054073,"kb_avail":261487028,"statfs":{"total":289910292480,"available":267762716672,"internally_reserved":0,"allocated":16970600448,"data_stored":20370413290,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1579788,"internal_metadata":5175370996},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,4,8,13,19,13,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22425600,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5564,"num_read_kb":22172,"num_write":5525,"num_write_kb":22100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001914"},"pg_stats":[{"pgid":"2.7","version":"109'13653","reported_seq":19270,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:04:23.818711+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:04:23.818711+0000","last_peered":"2023-10-17T02:04:23.818711+0000","last_clean":"2023-10-17T02:04:23.818711+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:04:23.818711+0000","last_undegraded":"2023-10-17T02:04:23.818711+0000","last_fullsized":"2023-10-17T02:04:23.818711+0000","mapping_epoch":13,"log_start":"75'3600","ondisk_log_start":"75'3600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"109'10285","reported_seq":15672,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:04:23.812534+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:04:23.812534+0000","last_peered":"2023-10-17T02:04:23.812534+0000","last_clean":"2023-10-17T02:04:23.812534+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:04:23.812534+0000","last_undegraded":"2023-10-17T02:04:23.812534+0000","last_fullsized":"2023-10-17T02:04:23.812534+0000","mapping_epoch":13,"log_start":"42'200","ondisk_log_start":"42'200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"109'6894","reported_seq":9793,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:04:23.795391+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:04:23.795391+0000","last_peered":"2023-10-17T02:04:23.795391+0000","last_clean":"2023-10-17T02:04:23.795391+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:04:23.795391+0000","last_undegraded":"2023-10-17T02:04:23.795391+0000","last_fullsized":"2023-10-17T02:04:23.795391+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6894,"log_dups_size":0,"ondisk_log_size":6894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"109'10318","reported_seq":15114,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:04:23.810809+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:04:23.810809+0000","last_peered":"2023-10-17T02:04:23.810809+0000","last_clean":"2023-10-17T02:04:23.810809+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:04:23.810809+0000","last_undegraded":"2023-10-17T02:04:23.810809+0000","last_fullsized":"2023-10-17T02:04:23.810809+0000","mapping_epoch":13,"log_start":"42'300","ondisk_log_start":"42'300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"109'528116","reported_seq":1052419,"reported_epoch":109,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:04:27.041201+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T02:04:27.041201+0000","last_peered":"2023-10-17T02:04:27.041201+0000","last_clean":"2023-10-17T02:04:27.041201+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:04:27.041201+0000","last_undegraded":"2023-10-17T02:04:27.041201+0000","last_fullsized":"2023-10-17T02:04:27.041201+0000","mapping_epoch":16,"log_start":"107'518110","ondisk_log_start":"107'518110","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2468,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 973s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7132459008,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":525014,"num_read_kb":2067788,"num_write":525425,"num_write_kb":2101700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":266,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-17T02:03:56.846257+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:03:56.846257+0000","last_peered":"2023-10-17T02:03:56.846257+0000","last_clean":"2023-10-17T02:03:56.846257+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:03:56.846257+0000","last_undegraded":"2023-10-17T02:03:56.846257+0000","last_fullsized":"2023-10-17T02:03:56.846257+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"109'507834","reported_seq":1013162,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:04:27.279331+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:04:27.279331+0000","last_peered":"2023-10-17T02:04:27.279331+0000","last_clean":"2023-10-17T02:04:27.279331+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:04:27.279331+0000","last_undegraded":"2023-10-17T02:04:27.279331+0000","last_fullsized":"2023-10-17T02:04:27.279331+0000","mapping_epoch":16,"log_start":"107'497819","ondisk_log_start":"107'497819","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6857019392,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":506016,"num_read_kb":1994884,"num_write":505184,"num_write_kb":2020736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2861,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:04:23.467178+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:04:23.467178+0000","last_peered":"2023-10-17T02:04:23.467178+0000","last_clean":"2023-10-17T02:04:23.467178+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:04:23.467178+0000","last_undegraded":"2023-10-17T02:04:23.467178+0000","last_fullsized":"2023-10-17T02:04:23.467178+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"109'529202","reported_seq":1054950,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:04:27.033170+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:04:27.033170+0000","last_peered":"2023-10-17T02:04:27.033170+0000","last_clean":"2023-10-17T02:04:27.033170+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:04:27.033170+0000","last_undegraded":"2023-10-17T02:04:27.033170+0000","last_fullsized":"2023-10-17T02:04:27.033170+0000","mapping_epoch":16,"log_start":"107'519125","ondisk_log_start":"107'519125","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7149076480,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":526478,"num_read_kb":2074388,"num_write":526448,"num_write_kb":2105792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4289,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:04:23.425127+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:04:23.425127+0000","last_peered":"2023-10-17T02:04:23.425127+0000","last_clean":"2023-10-17T02:04:23.425127+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:04:23.425127+0000","last_undegraded":"2023-10-17T02:04:23.425127+0000","last_fullsized":"2023-10-17T02:04:23.425127+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"109'516446","reported_seq":1031929,"reported_epoch":109,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:04:27.303901+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:04:27.303901+0000","last_peered":"2023-10-17T02:04:27.303901+0000","last_clean":"2023-10-17T02:04:27.303901+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:04:27.303901+0000","last_undegraded":"2023-10-17T02:04:27.303901+0000","last_fullsized":"2023-10-17T02:04:27.303901+0000","mapping_epoch":16,"log_start":"107'506416","ondisk_log_start":"107'506416","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":512,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 181s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":6994399251,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":516240,"num_read_kb":2033556,"num_write":514893,"num_write_kb":2059566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"109'10288","reported_seq":14449,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:04:23.863202+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:04:23.863202+0000","last_peered":"2023-10-17T02:04:23.863202+0000","last_clean":"2023-10-17T02:04:23.863202+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:04:23.863202+0000","last_undegraded":"2023-10-17T02:04:23.863202+0000","last_fullsized":"2023-10-17T02:04:23.863202+0000","mapping_epoch":13,"log_start":"42'200","ondisk_log_start":"42'200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":286,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-17T02:02:52.335311+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:02:52.335311+0000","last_peered":"2023-10-17T02:02:52.335311+0000","last_clean":"2023-10-17T02:02:52.335311+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:02:52.335311+0000","last_undegraded":"2023-10-17T02:02:52.335311+0000","last_fullsized":"2023-10-17T02:02:52.335311+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28132954131,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2073748,"num_read_kb":8170616,"num_write":2071950,"num_write_kb":8287794,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16969367552,"data_stored":20370227238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40128,"ondisk_log_size":40128,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":47366,"ondisk_log_size":47366,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739359,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10286684,"kb_used_data":8283924,"kb_used_omap":764,"kb_used_meta":2001987,"kb_avail":84085156,"statfs":{"total":96636764160,"available":86103199744,"internally_reserved":0,"allocated":8482738176,"data_stored":10182544887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":782639,"internal_metadata":2050035409},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,3,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:03:25 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.41999999999999998,"15min":0.41299999999999998},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":1.3660000000000001,"5min":8.4190000000000005,"15min":8.4190000000000005},"last":3.8879999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51200000000000001,"15min":0.49299999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.188},"max":{"1min":3.629,"5min":8.3559999999999999,"15min":8.3559999999999999},"last":3.8439999999999999}]},{"osd":1,"last update":"Tue Oct 17 02:03:59 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.41299999999999998,"15min":0.48099999999999998},"min":{"1min":0.23999999999999999,"5min":0.218,"15min":0.20699999999999999},"max":{"1min":6.202,"5min":6.202,"15min":9.5429999999999993},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.54100000000000004,"15min":0.57599999999999996},"min":{"1min":0.26100000000000001,"5min":0.23200000000000001,"15min":0.19500000000000001},"max":{"1min":6.1699999999999999,"5min":8.3930000000000007,"15min":9.5999999999999996},"last":0.35399999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739356,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5790388,"kb_used_data":4080876,"kb_used_omap":362,"kb_used_meta":1709141,"kb_avail":88581452,"statfs":{"total":96636764160,"available":90707406848,"internally_reserved":0,"allocated":4178817024,"data_stored":5016397334,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":371063,"internal_metadata":1750161033},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,8,6,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:04:17 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.45600000000000002,"15min":0.47699999999999998},"min":{"1min":0.17799999999999999,"5min":0.16600000000000001,"15min":0.16300000000000001},"max":{"1min":2.2330000000000001,"5min":5.5869999999999997,"15min":6.6890000000000001},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.50800000000000001,"15min":0.52800000000000002},"min":{"1min":0.251,"5min":0.23000000000000001,"15min":0.22},"max":{"1min":2.262,"5min":5.6360000000000001,"15min":6.7539999999999996},"last":0.308}]},{"osd":2,"last update":"Tue Oct 17 02:04:17 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.624,"15min":0.53400000000000003},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.188},"max":{"1min":5.2880000000000003,"5min":6.8230000000000004,"15min":6.8230000000000004},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.54900000000000004,"15min":0.501},"min":{"1min":0.23300000000000001,"5min":0.22600000000000001,"15min":0.20100000000000001},"max":{"1min":5.2089999999999996,"5min":6.7800000000000002,"15min":6.7800000000000002},"last":0.28899999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739356,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5551420,"kb_used_data":4208052,"kb_used_omap":416,"kb_used_meta":1342943,"kb_avail":88820420,"statfs":{"total":96636764160,"available":90952110080,"internally_reserved":0,"allocated":4309045248,"data_stored":5171471069,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":426086,"internal_metadata":1375174554},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,3,9,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:03:52 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.46000000000000002,"15min":0.42199999999999999},"min":{"1min":0.19,"5min":0.17999999999999999,"15min":0.17000000000000001},"max":{"1min":1.675,"5min":4.101,"15min":5.9829999999999997},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.40500000000000003,"15min":0.48199999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":6.3460000000000001,"5min":6.3460000000000001,"15min":6.3460000000000001},"last":0.32400000000000001}]},{"osd":2,"last update":"Tue Oct 17 02:04:01 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.53100000000000003,"15min":0.42599999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":1.3340000000000001,"5min":6.0010000000000003,"15min":6.0010000000000003},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.60699999999999998,"15min":0.61199999999999999},"min":{"1min":0.23699999999999999,"5min":0.23100000000000001,"15min":0.19400000000000001},"max":{"1min":6.3899999999999997,"5min":8.0169999999999995,"15min":10.378},"last":0.29399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4308484096,"data_stored":5171081216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4178243584,"data_stored":5016076307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8482639872,"data_stored":10183069715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:04:28.258 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:04:28.259 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:04:28.259 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:04:48.260 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:04:48.817 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:04:48.818 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:04:48.873 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2587,"stamp":"2023-10-17T02:04:47.528329+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28162927448,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2104973,"num_read_kb":8220181,"num_write":2079839,"num_write_kb":8319007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87556,"ondisk_log_size":87556,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21204832,"kb_used_data":16631104,"kb_used_omap":1544,"kb_used_meta":4572151,"kb_avail":261910688,"statfs":{"total":289910292480,"available":268196544512,"internally_reserved":0,"allocated":17030250496,"data_stored":20430063338,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1581468,"internal_metadata":4681883236},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,6,12,9,25,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22347776,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5561,"num_read_kb":22136,"num_write":5517,"num_write_kb":22068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002228"},"pg_stats":[{"pgid":"2.7","version":"109'13689","reported_seq":19322,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:04:45.468197+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:04:45.468197+0000","last_peered":"2023-10-17T02:04:45.468197+0000","last_clean":"2023-10-17T02:04:45.468197+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:04:45.468197+0000","last_undegraded":"2023-10-17T02:04:45.468197+0000","last_fullsized":"2023-10-17T02:04:45.468197+0000","mapping_epoch":13,"log_start":"75'3600","ondisk_log_start":"75'3600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"109'10312","reported_seq":15711,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:04:45.461751+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:04:45.461751+0000","last_peered":"2023-10-17T02:04:45.461751+0000","last_clean":"2023-10-17T02:04:45.461751+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:04:45.461751+0000","last_undegraded":"2023-10-17T02:04:45.461751+0000","last_fullsized":"2023-10-17T02:04:45.461751+0000","mapping_epoch":13,"log_start":"42'300","ondisk_log_start":"42'300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"109'6912","reported_seq":9819,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:04:45.451352+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:04:45.451352+0000","last_peered":"2023-10-17T02:04:45.451352+0000","last_clean":"2023-10-17T02:04:45.451352+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:04:45.451352+0000","last_undegraded":"2023-10-17T02:04:45.451352+0000","last_fullsized":"2023-10-17T02:04:45.451352+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6912,"log_dups_size":0,"ondisk_log_size":6912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"109'10345","reported_seq":15153,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:04:45.466917+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:04:45.466917+0000","last_peered":"2023-10-17T02:04:45.466917+0000","last_clean":"2023-10-17T02:04:45.466917+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:04:45.466917+0000","last_undegraded":"2023-10-17T02:04:45.466917+0000","last_fullsized":"2023-10-17T02:04:45.466917+0000","mapping_epoch":13,"log_start":"42'300","ondisk_log_start":"42'300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"109'530020","reported_seq":1056202,"reported_epoch":109,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:04:47.153049+0000","last_change":"2023-10-17T01:48:12.867633+0000","last_active":"2023-10-17T02:04:47.153049+0000","last_peered":"2023-10-17T02:04:47.153049+0000","last_clean":"2023-10-17T02:04:47.153049+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:04:47.153049+0000","last_undegraded":"2023-10-17T02:04:47.153049+0000","last_fullsized":"2023-10-17T02:04:47.153049+0000","mapping_epoch":16,"log_start":"108'520010","ondisk_log_start":"108'520010","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2524,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 993s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7140048896,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":526885,"num_read_kb":2075232,"num_write":527300,"num_write_kb":2109200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":266,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-17T02:03:56.846257+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:03:56.846257+0000","last_peered":"2023-10-17T02:03:56.846257+0000","last_clean":"2023-10-17T02:03:56.846257+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:03:56.846257+0000","last_undegraded":"2023-10-17T02:03:56.846257+0000","last_fullsized":"2023-10-17T02:03:56.846257+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"109'509650","reported_seq":1016695,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:04:47.396561+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:04:47.396561+0000","last_peered":"2023-10-17T02:04:47.396561+0000","last_clean":"2023-10-17T02:04:47.396561+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:04:47.396561+0000","last_undegraded":"2023-10-17T02:04:47.396561+0000","last_fullsized":"2023-10-17T02:04:47.396561+0000","mapping_epoch":16,"log_start":"108'499619","ondisk_log_start":"108'499619","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6864367616,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":507730,"num_read_kb":2001724,"num_write":507000,"num_write_kb":2028000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2869,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:04:43.907962+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:04:43.907962+0000","last_peered":"2023-10-17T02:04:43.907962+0000","last_clean":"2023-10-17T02:04:43.907962+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:04:43.907962+0000","last_undegraded":"2023-10-17T02:04:43.907962+0000","last_fullsized":"2023-10-17T02:04:43.907962+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"109'530995","reported_seq":1058659,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:04:47.143838+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:04:47.143838+0000","last_peered":"2023-10-17T02:04:47.143838+0000","last_clean":"2023-10-17T02:04:47.143838+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:04:47.143838+0000","last_undegraded":"2023-10-17T02:04:47.143838+0000","last_fullsized":"2023-10-17T02:04:47.143838+0000","mapping_epoch":16,"log_start":"108'520925","ondisk_log_start":"108'520925","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7156314112,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":528384,"num_read_kb":2081960,"num_write":528241,"num_write_kb":2112964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4302,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:04:43.907575+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:04:43.907575+0000","last_peered":"2023-10-17T02:04:43.907575+0000","last_clean":"2023-10-17T02:04:43.907575+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:04:43.907575+0000","last_undegraded":"2023-10-17T02:04:43.907575+0000","last_fullsized":"2023-10-17T02:04:43.907575+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"109'518262","reported_seq":1035522,"reported_epoch":109,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:04:47.386285+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:04:47.386285+0000","last_peered":"2023-10-17T02:04:47.386285+0000","last_clean":"2023-10-17T02:04:47.386285+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:04:47.386285+0000","last_undegraded":"2023-10-17T02:04:47.386285+0000","last_fullsized":"2023-10-17T02:04:47.386285+0000","mapping_epoch":16,"log_start":"108'508217","ondisk_log_start":"108'508217","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":580,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 201s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7001604115,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":518012,"num_read_kb":2040612,"num_write":516668,"num_write_kb":2066666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"109'10315","reported_seq":14488,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:04:45.469316+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:04:45.469316+0000","last_peered":"2023-10-17T02:04:45.469316+0000","last_clean":"2023-10-17T02:04:45.469316+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:04:45.469316+0000","last_undegraded":"2023-10-17T02:04:45.469316+0000","last_fullsized":"2023-10-17T02:04:45.469316+0000","mapping_epoch":13,"log_start":"42'300","ondisk_log_start":"42'300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":287,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:04:02.349554+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:04:02.349554+0000","last_peered":"2023-10-17T02:04:02.349554+0000","last_clean":"2023-10-17T02:04:02.349554+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:04:02.349554+0000","last_undegraded":"2023-10-17T02:04:02.349554+0000","last_fullsized":"2023-10-17T02:04:02.349554+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28162334739,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2081011,"num_read_kb":8199528,"num_write":2079209,"num_write_kb":8316830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17028677632,"data_stored":20429537318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40156,"ondisk_log_size":40156,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":47301,"ondisk_log_size":47301,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739363,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9729352,"kb_used_data":8312568,"kb_used_omap":765,"kb_used_meta":1416002,"kb_avail":84642488,"statfs":{"total":96636764160,"available":86673907712,"internally_reserved":0,"allocated":8512069632,"data_stored":10211876343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":784319,"internal_metadata":1449986113},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:04:26 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.45200000000000001,"15min":0.41699999999999998},"min":{"1min":0.214,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":3.8879999999999999,"5min":8.4190000000000005,"15min":8.4190000000000005},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.52800000000000002,"15min":0.504},"min":{"1min":0.25,"5min":0.192,"15min":0.188},"max":{"1min":3.8439999999999999,"5min":8.3559999999999999,"15min":8.3559999999999999},"last":0.39800000000000002}]},{"osd":1,"last update":"Tue Oct 17 02:03:59 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.41299999999999998,"15min":0.48099999999999998},"min":{"1min":0.23999999999999999,"5min":0.218,"15min":0.20699999999999999},"max":{"1min":6.202,"5min":6.202,"15min":9.5429999999999993},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.54100000000000004,"15min":0.57599999999999996},"min":{"1min":0.26100000000000001,"5min":0.23200000000000001,"15min":0.19500000000000001},"max":{"1min":6.1699999999999999,"5min":8.3930000000000007,"15min":9.5999999999999996},"last":0.34399999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739360,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5855384,"kb_used_data":4095888,"kb_used_omap":362,"kb_used_meta":1759125,"kb_avail":88516456,"statfs":{"total":96636764160,"available":90640850944,"internally_reserved":0,"allocated":4194189312,"data_stored":5031769622,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":371063,"internal_metadata":1801344649},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,3,1,15,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:04:17 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.45600000000000002,"15min":0.47699999999999998},"min":{"1min":0.17799999999999999,"5min":0.16600000000000001,"15min":0.16300000000000001},"max":{"1min":2.2330000000000001,"5min":5.5869999999999997,"15min":6.6890000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.50800000000000001,"15min":0.52800000000000002},"min":{"1min":0.251,"5min":0.23000000000000001,"15min":0.22},"max":{"1min":2.262,"5min":5.6360000000000001,"15min":6.7539999999999996},"last":0.63500000000000001}]},{"osd":2,"last update":"Tue Oct 17 02:04:17 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.624,"15min":0.53400000000000003},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.188},"max":{"1min":5.2880000000000003,"5min":6.8230000000000004,"15min":6.8230000000000004},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.54900000000000004,"15min":0.501},"min":{"1min":0.23300000000000001,"5min":0.22600000000000001,"15min":0.20100000000000001},"max":{"1min":5.2089999999999996,"5min":6.7800000000000002,"15min":6.7800000000000002},"last":0.45000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739360,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5620096,"kb_used_data":4222648,"kb_used_omap":416,"kb_used_meta":1397023,"kb_avail":88751744,"statfs":{"total":96636764160,"available":90881785856,"internally_reserved":0,"allocated":4323991552,"data_stored":5186417373,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":426086,"internal_metadata":1430552474},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,6,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:03:52 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.46000000000000002,"15min":0.42199999999999999},"min":{"1min":0.19,"5min":0.17999999999999999,"15min":0.17000000000000001},"max":{"1min":1.675,"5min":4.101,"15min":5.9829999999999997},"last":0.19700000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.40500000000000003,"15min":0.48199999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":6.3460000000000001,"5min":6.3460000000000001,"15min":6.3460000000000001},"last":0.29099999999999998}]},{"osd":2,"last update":"Tue Oct 17 02:04:01 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.53100000000000003,"15min":0.42599999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":1.3340000000000001,"5min":6.0010000000000003,"15min":6.0010000000000003},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.60699999999999998,"15min":0.61199999999999999},"min":{"1min":0.23699999999999999,"5min":0.23100000000000001,"15min":0.19400000000000001},"max":{"1min":6.3899999999999997,"5min":8.0169999999999995,"15min":10.378},"last":0.32900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4323500032,"data_stored":5186097152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4192940032,"data_stored":5030772755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8512237568,"data_stored":10212667411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:04:48.875 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:04:48.875 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:04:48.876 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:05:04.407 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:04.399+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T02:05:04.440 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:04.435+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T02:05:04.441 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:04.435+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbda750 2023-10-17T02:05:04.441 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:04.435+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T02:05:04.441 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:04.435+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T02:05:04.442 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:04.435+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T02:05:04.442 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:04.435+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T02:05:04.442 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:04.435+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T02:05:04.443 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:04.435+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T02:05:04.443 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:04.435+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T02:05:04.443 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:04.435+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T02:05:04.443 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:04.435+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T02:05:04.444 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:04.435+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T02:05:04.444 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:04.435+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T02:05:04.444 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:04.435+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T02:05:08.877 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:05:09.464 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:05:09.465 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:05:09.493 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2597,"stamp":"2023-10-17T02:05:07.531666+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28188642136,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2111366,"num_read_kb":8245633,"num_write":2086170,"num_write_kb":8344331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87639,"ondisk_log_size":87639,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21425708,"kb_used_data":16686544,"kb_used_omap":1544,"kb_used_meta":4737591,"kb_avail":261689812,"statfs":{"total":289910292480,"available":267970367488,"internally_reserved":0,"allocated":17087021056,"data_stored":20486833898,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1581468,"internal_metadata":4851293796},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,6,9,11,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18497536,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4595,"num_read_kb":18304,"num_write":4540,"num_write_kb":18160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002035"},"pg_stats":[{"pgid":"2.7","version":"109'13741","reported_seq":19387,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:05:01.248932+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:05:01.248932+0000","last_peered":"2023-10-17T02:05:01.248932+0000","last_clean":"2023-10-17T02:05:01.248932+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:05:01.248932+0000","last_undegraded":"2023-10-17T02:05:01.248932+0000","last_fullsized":"2023-10-17T02:05:01.248932+0000","mapping_epoch":13,"log_start":"75'3700","ondisk_log_start":"75'3700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"109'10351","reported_seq":15760,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:05:01.248180+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:05:01.248180+0000","last_peered":"2023-10-17T02:05:01.248180+0000","last_clean":"2023-10-17T02:05:01.248180+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:05:01.248180+0000","last_undegraded":"2023-10-17T02:05:01.248180+0000","last_fullsized":"2023-10-17T02:05:01.248180+0000","mapping_epoch":13,"log_start":"42'300","ondisk_log_start":"42'300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"109'6938","reported_seq":9851,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:05:01.227023+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:05:01.227023+0000","last_peered":"2023-10-17T02:05:01.227023+0000","last_clean":"2023-10-17T02:05:01.227023+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:05:01.227023+0000","last_undegraded":"2023-10-17T02:05:01.227023+0000","last_fullsized":"2023-10-17T02:05:01.227023+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6938,"log_dups_size":0,"ondisk_log_size":6938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"109'10384","reported_seq":15201,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:05:01.242170+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:05:01.242170+0000","last_peered":"2023-10-17T02:05:01.242170+0000","last_clean":"2023-10-17T02:05:01.242170+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:05:01.242170+0000","last_undegraded":"2023-10-17T02:05:01.242170+0000","last_fullsized":"2023-10-17T02:05:01.242170+0000","mapping_epoch":13,"log_start":"42'300","ondisk_log_start":"42'300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"109'531953","reported_seq":1059946,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:05:07.260335+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:05:07.260335+0000","last_peered":"2023-10-17T02:05:07.260335+0000","last_clean":"2023-10-17T02:05:07.260335+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:05:07.260335+0000","last_undegraded":"2023-10-17T02:05:07.260335+0000","last_fullsized":"2023-10-17T02:05:07.260335+0000","mapping_epoch":16,"log_start":"108'521910","ondisk_log_start":"108'521910","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7147728896,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":528688,"num_read_kb":2082412,"num_write":529196,"num_write_kb":2116784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":266,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-17T02:03:56.846257+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:03:56.846257+0000","last_peered":"2023-10-17T02:03:56.846257+0000","last_clean":"2023-10-17T02:03:56.846257+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:03:56.846257+0000","last_undegraded":"2023-10-17T02:03:56.846257+0000","last_fullsized":"2023-10-17T02:03:56.846257+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"109'510925","reported_seq":1019270,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:05:02.465356+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:05:02.465356+0000","last_peered":"2023-10-17T02:05:02.465356+0000","last_clean":"2023-10-17T02:05:02.465356+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:05:02.465356+0000","last_undegraded":"2023-10-17T02:05:02.465356+0000","last_fullsized":"2023-10-17T02:05:02.465356+0000","mapping_epoch":16,"log_start":"108'500919","ondisk_log_start":"108'500919","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6869536768,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":509034,"num_read_kb":2006920,"num_write":508275,"num_write_kb":2033100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2877,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:05:04.387144+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:05:04.387144+0000","last_peered":"2023-10-17T02:05:04.387144+0000","last_clean":"2023-10-17T02:05:04.387144+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:05:04.387144+0000","last_undegraded":"2023-10-17T02:05:04.387144+0000","last_fullsized":"2023-10-17T02:05:04.387144+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"109'532781","reported_seq":1062389,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:05:07.272555+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:05:07.272555+0000","last_peered":"2023-10-17T02:05:07.272555+0000","last_clean":"2023-10-17T02:05:07.272555+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:05:07.272555+0000","last_undegraded":"2023-10-17T02:05:07.272555+0000","last_fullsized":"2023-10-17T02:05:07.272555+0000","mapping_epoch":16,"log_start":"108'522725","ondisk_log_start":"108'522725","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7163592704,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":530329,"num_read_kb":2089700,"num_write":530027,"num_write_kb":2120108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4314,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:05:04.389970+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:05:04.389970+0000","last_peered":"2023-10-17T02:05:04.389970+0000","last_clean":"2023-10-17T02:05:04.389970+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:05:04.389970+0000","last_undegraded":"2023-10-17T02:05:04.389970+0000","last_fullsized":"2023-10-17T02:05:04.389970+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"109'519656","reported_seq":1038268,"reported_epoch":109,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:05:02.473652+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:05:02.473652+0000","last_peered":"2023-10-17T02:05:02.473652+0000","last_clean":"2023-10-17T02:05:02.473652+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:05:02.473652+0000","last_undegraded":"2023-10-17T02:05:02.473652+0000","last_fullsized":"2023-10-17T02:05:02.473652+0000","mapping_epoch":16,"log_start":"108'509617","ondisk_log_start":"108'509617","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":618,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 217s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7007191059,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":519353,"num_read_kb":2045948,"num_write":518042,"num_write_kb":2072162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"109'10354","reported_seq":14536,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:05:01.297020+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:05:01.297020+0000","last_peered":"2023-10-17T02:05:01.297020+0000","last_clean":"2023-10-17T02:05:01.297020+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:05:01.297020+0000","last_undegraded":"2023-10-17T02:05:01.297020+0000","last_fullsized":"2023-10-17T02:05:01.297020+0000","mapping_epoch":13,"log_start":"42'300","ondisk_log_start":"42'300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":287,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:04:02.349554+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:04:02.349554+0000","last_peered":"2023-10-17T02:04:02.349554+0000","last_clean":"2023-10-17T02:04:02.349554+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:04:02.349554+0000","last_undegraded":"2023-10-17T02:04:02.349554+0000","last_fullsized":"2023-10-17T02:04:02.349554+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28188049427,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2087404,"num_read_kb":8224980,"num_write":2085540,"num_write_kb":8342154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17084461056,"data_stored":20485320742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40144,"ondisk_log_size":40144,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":47396,"ondisk_log_size":47396,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739367,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9856952,"kb_used_data":8342768,"kb_used_omap":765,"kb_used_meta":1513410,"kb_avail":84514888,"statfs":{"total":96636764160,"available":86543245312,"internally_reserved":0,"allocated":8542994432,"data_stored":10242801143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":784319,"internal_metadata":1549731905},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,5,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:04:26 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.45200000000000001,"15min":0.41699999999999998},"min":{"1min":0.214,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":3.8879999999999999,"5min":8.4190000000000005,"15min":8.4190000000000005},"last":0.188},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.52800000000000002,"15min":0.504},"min":{"1min":0.25,"5min":0.192,"15min":0.188},"max":{"1min":3.8439999999999999,"5min":8.3559999999999999,"15min":8.3559999999999999},"last":0.27500000000000002}]},{"osd":1,"last update":"Tue Oct 17 02:05:00 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.439,"15min":0.48899999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.20699999999999999},"max":{"1min":3.7839999999999998,"5min":6.202,"15min":9.5429999999999993},"last":0.35899999999999999},{"interface":"front","average":{"1min":1.032,"5min":0.67000000000000004,"15min":0.623},"min":{"1min":0.27700000000000002,"5min":0.23200000000000001,"15min":0.19500000000000001},"max":{"1min":12.824999999999999,"5min":12.824999999999999,"15min":12.824999999999999},"last":0.38700000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739363,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5897908,"kb_used_data":4106472,"kb_used_omap":362,"kb_used_meta":1791061,"kb_avail":88473932,"statfs":{"total":96636764160,"available":90597306368,"internally_reserved":0,"allocated":4205027328,"data_stored":5042607638,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":371063,"internal_metadata":1834047113},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,4,5,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:04:17 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.45600000000000002,"15min":0.47699999999999998},"min":{"1min":0.17799999999999999,"5min":0.16600000000000001,"15min":0.16300000000000001},"max":{"1min":2.2330000000000001,"5min":5.5869999999999997,"15min":6.6890000000000001},"last":0.19800000000000001},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.50800000000000001,"15min":0.52800000000000002},"min":{"1min":0.251,"5min":0.23000000000000001,"15min":0.22},"max":{"1min":2.262,"5min":5.6360000000000001,"15min":6.7539999999999996},"last":0.26600000000000001}]},{"osd":2,"last update":"Tue Oct 17 02:04:17 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.624,"15min":0.53400000000000003},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.188},"max":{"1min":5.2880000000000003,"5min":6.8230000000000004,"15min":6.8230000000000004},"last":0.248},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.54900000000000004,"15min":0.501},"min":{"1min":0.23300000000000001,"5min":0.22600000000000001,"15min":0.20100000000000001},"max":{"1min":5.2089999999999996,"5min":6.7800000000000002,"15min":6.7800000000000002},"last":0.29899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739364,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5670848,"kb_used_data":4237304,"kb_used_omap":416,"kb_used_meta":1433119,"kb_avail":88700992,"statfs":{"total":96636764160,"available":90829815808,"internally_reserved":0,"allocated":4338999296,"data_stored":5201425117,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":426086,"internal_metadata":1467514778},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:04:55 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.47899999999999998,"15min":0.41499999999999998},"min":{"1min":0.184,"5min":0.17999999999999999,"15min":0.17000000000000001},"max":{"1min":3.282,"5min":4.101,"15min":4.1680000000000001},"last":0.20699999999999999},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.52100000000000002,"15min":0.497},"min":{"1min":0.20799999999999999,"5min":0.191,"15min":0.191},"max":{"1min":7.4880000000000004,"5min":7.4880000000000004,"15min":7.4880000000000004},"last":0.23499999999999999}]},{"osd":2,"last update":"Tue Oct 17 02:05:05 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.59799999999999998,"15min":0.45700000000000002},"min":{"1min":0.23100000000000001,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":4.6470000000000002,"5min":6.0010000000000003,"15min":6.0010000000000003},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.59599999999999997,"15min":0.60899999999999999},"min":{"1min":0.25800000000000001,"5min":0.23100000000000001,"15min":0.19400000000000001},"max":{"1min":2.8959999999999999,"5min":8.0169999999999995,"15min":10.378},"last":0.27400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4338593792,"data_stored":5201190912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4203814912,"data_stored":5041647635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8542052352,"data_stored":10242482195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:05:09.495 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:05:09.496 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:05:11.040 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:11.031+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T02:05:11.040 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:11.031+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T02:05:11.053 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:11.047+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T02:05:11.063 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:11.055+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T02:05:15.252 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:15.247+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T02:05:15.253 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:15.247+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T02:05:15.253 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:15.247+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T02:05:15.253 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:15.247+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T02:05:15.253 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:15.247+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T02:05:15.258 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:15.251+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T02:05:23.074 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:23.067+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T02:05:23.081 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:23.075+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T02:05:23.101 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:23.095+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T02:05:29.496 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:05:30.023 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:05:30.024 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:05:30.051 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2609,"stamp":"2023-10-17T02:05:29.546548+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28222589784,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2119458,"num_read_kb":8277829,"num_write":2094492,"num_write_kb":8377619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87629,"ondisk_log_size":87629,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21739812,"kb_used_data":16749640,"kb_used_omap":1544,"kb_used_meta":4988599,"kb_avail":261375708,"statfs":{"total":289910292480,"available":267648724992,"internally_reserved":0,"allocated":17151631360,"data_stored":20551430870,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1581468,"internal_metadata":5108325988},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[17,1,0,3,7,25,19,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15810560,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3733,"num_read_kb":14872,"num_write":3879,"num_write_kb":15516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.012803"},"pg_stats":[{"pgid":"2.7","version":"110'13828","reported_seq":19493,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:05:27.564232+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:05:27.564232+0000","last_peered":"2023-10-17T02:05:27.564232+0000","last_clean":"2023-10-17T02:05:27.564232+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:05:27.564232+0000","last_undegraded":"2023-10-17T02:05:27.564232+0000","last_fullsized":"2023-10-17T02:05:27.564232+0000","mapping_epoch":13,"log_start":"77'3800","ondisk_log_start":"77'3800","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"110'10416","reported_seq":15839,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:05:27.578803+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:05:27.578803+0000","last_peered":"2023-10-17T02:05:27.578803+0000","last_clean":"2023-10-17T02:05:27.578803+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:05:27.578803+0000","last_undegraded":"2023-10-17T02:05:27.578803+0000","last_fullsized":"2023-10-17T02:05:27.578803+0000","mapping_epoch":13,"log_start":"44'400","ondisk_log_start":"44'400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"110'6983","reported_seq":9906,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:05:27.550950+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:05:27.550950+0000","last_peered":"2023-10-17T02:05:27.550950+0000","last_clean":"2023-10-17T02:05:27.550950+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:05:27.550950+0000","last_undegraded":"2023-10-17T02:05:27.550950+0000","last_fullsized":"2023-10-17T02:05:27.550950+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":6983,"log_dups_size":0,"ondisk_log_size":6983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"110'10451","reported_seq":15283,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:05:27.588489+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:05:27.588489+0000","last_peered":"2023-10-17T02:05:27.588489+0000","last_clean":"2023-10-17T02:05:27.588489+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:05:27.588489+0000","last_undegraded":"2023-10-17T02:05:27.588489+0000","last_fullsized":"2023-10-17T02:05:27.588489+0000","mapping_epoch":13,"log_start":"44'400","ondisk_log_start":"44'400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"110'533742","reported_seq":1063533,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:05:27.349083+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:05:27.349083+0000","last_peered":"2023-10-17T02:05:27.349083+0000","last_clean":"2023-10-17T02:05:27.349083+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:05:27.349083+0000","last_undegraded":"2023-10-17T02:05:27.349083+0000","last_fullsized":"2023-10-17T02:05:27.349083+0000","mapping_epoch":16,"log_start":"108'523710","ondisk_log_start":"108'523710","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7154982912,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":530477,"num_read_kb":2089536,"num_write":530985,"num_write_kb":2123940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":267,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:04:02.342251+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:04:02.342251+0000","last_peered":"2023-10-17T02:04:02.342251+0000","last_clean":"2023-10-17T02:04:02.342251+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:04:02.342251+0000","last_undegraded":"2023-10-17T02:04:02.342251+0000","last_fullsized":"2023-10-17T02:04:02.342251+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"110'513172","reported_seq":1023700,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:05:27.755306+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:05:27.755306+0000","last_peered":"2023-10-17T02:05:27.755306+0000","last_clean":"2023-10-17T02:05:27.755306+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:05:27.755306+0000","last_undegraded":"2023-10-17T02:05:27.755306+0000","last_fullsized":"2023-10-17T02:05:27.755306+0000","mapping_epoch":16,"log_start":"108'503119","ondisk_log_start":"108'503119","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6878703616,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":511212,"num_read_kb":2015572,"num_write":510522,"num_write_kb":2042088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2884,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:05:19.745490+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:05:19.745490+0000","last_peered":"2023-10-17T02:05:19.745490+0000","last_clean":"2023-10-17T02:05:19.745490+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:05:19.745490+0000","last_undegraded":"2023-10-17T02:05:19.745490+0000","last_fullsized":"2023-10-17T02:05:19.745490+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"110'534694","reported_seq":1066191,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:05:27.335024+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:05:27.335024+0000","last_peered":"2023-10-17T02:05:27.335024+0000","last_clean":"2023-10-17T02:05:27.335024+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:05:27.335024+0000","last_undegraded":"2023-10-17T02:05:27.335024+0000","last_fullsized":"2023-10-17T02:05:27.335024+0000","mapping_epoch":16,"log_start":"108'524625","ondisk_log_start":"108'524625","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7171416064,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":532220,"num_read_kb":2097228,"num_write":531940,"num_write_kb":2127760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4324,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:05:19.762618+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:05:19.762618+0000","last_peered":"2023-10-17T02:05:19.762618+0000","last_clean":"2023-10-17T02:05:19.762618+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:05:19.762618+0000","last_undegraded":"2023-10-17T02:05:19.762618+0000","last_fullsized":"2023-10-17T02:05:19.762618+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"110'522068","reported_seq":1042925,"reported_epoch":110,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:05:27.798649+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:05:27.798649+0000","last_peered":"2023-10-17T02:05:27.798649+0000","last_clean":"2023-10-17T02:05:27.798649+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:05:27.798649+0000","last_undegraded":"2023-10-17T02:05:27.798649+0000","last_fullsized":"2023-10-17T02:05:27.798649+0000","mapping_epoch":16,"log_start":"108'512018","ondisk_log_start":"108'512018","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":680,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 242s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7016894483,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":521587,"num_read_kb":2054840,"num_write":520415,"num_write_kb":2081654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"110'10420","reported_seq":14616,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:05:27.589591+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:05:27.589591+0000","last_peered":"2023-10-17T02:05:27.589591+0000","last_clean":"2023-10-17T02:05:27.589591+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:05:27.589591+0000","last_undegraded":"2023-10-17T02:05:27.589591+0000","last_fullsized":"2023-10-17T02:05:27.589591+0000","mapping_epoch":13,"log_start":"44'400","ondisk_log_start":"44'400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":288,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:04:42.362981+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:04:42.362981+0000","last_peered":"2023-10-17T02:04:42.362981+0000","last_clean":"2023-10-17T02:04:42.362981+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:04:42.362981+0000","last_undegraded":"2023-10-17T02:04:42.362981+0000","last_fullsized":"2023-10-17T02:04:42.362981+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28221997075,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2095496,"num_read_kb":8257176,"num_write":2093862,"num_write_kb":8375442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17148608512,"data_stored":20549468198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40204,"ondisk_log_size":40204,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":47326,"ondisk_log_size":47326,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739371,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9974452,"kb_used_data":8372136,"kb_used_omap":765,"kb_used_meta":1601538,"kb_avail":84397388,"statfs":{"total":96636764160,"available":86422925312,"internally_reserved":0,"allocated":8573067264,"data_stored":10272869531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":784319,"internal_metadata":1639974977},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,7,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:04:26 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.45200000000000001,"15min":0.41699999999999998},"min":{"1min":0.214,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":3.8879999999999999,"5min":8.4190000000000005,"15min":8.4190000000000005},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.52800000000000002,"15min":0.504},"min":{"1min":0.25,"5min":0.192,"15min":0.188},"max":{"1min":3.8439999999999999,"5min":8.3559999999999999,"15min":8.3559999999999999},"last":0.247}]},{"osd":1,"last update":"Tue Oct 17 02:05:00 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.439,"15min":0.48899999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.20699999999999999},"max":{"1min":3.7839999999999998,"5min":6.202,"15min":9.5429999999999993},"last":3.9020000000000001},{"interface":"front","average":{"1min":1.032,"5min":0.67000000000000004,"15min":0.623},"min":{"1min":0.27700000000000002,"5min":0.23200000000000001,"15min":0.19500000000000001},"max":{"1min":12.824999999999999,"5min":12.824999999999999,"15min":12.824999999999999},"last":3.9350000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739368,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5999512,"kb_used_data":4124688,"kb_used_omap":362,"kb_used_meta":1874453,"kb_avail":88372328,"statfs":{"total":96636764160,"available":90493263872,"internally_reserved":0,"allocated":4223680512,"data_stored":5061256378,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":371063,"internal_metadata":1919440521},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[16,1,0,2,3,12,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:05:21 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.54600000000000004,"15min":0.51000000000000001},"min":{"1min":0.17999999999999999,"5min":0.16600000000000001,"15min":0.16300000000000001},"max":{"1min":6.8049999999999997,"5min":6.8049999999999997,"15min":6.8049999999999997},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.59599999999999997,"15min":0.56000000000000005},"min":{"1min":0.24299999999999999,"5min":0.23000000000000001,"15min":0.22},"max":{"1min":6.8479999999999999,"5min":6.8479999999999999,"15min":6.8479999999999999},"last":0.38100000000000001}]},{"osd":2,"last update":"Tue Oct 17 02:05:21 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.60599999999999998,"15min":0.53300000000000003},"min":{"1min":0.216,"5min":0.19500000000000001,"15min":0.188},"max":{"1min":0.92800000000000005,"5min":6.8230000000000004,"15min":6.8230000000000004},"last":25.128},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.53900000000000003,"15min":0.499},"min":{"1min":0.249,"5min":0.22600000000000001,"15min":0.20100000000000001},"max":{"1min":0.89300000000000002,"5min":6.7800000000000002,"15min":6.7800000000000002},"last":0.24399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739368,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5765848,"kb_used_data":4252816,"kb_used_omap":416,"kb_used_meta":1512607,"kb_avail":88605992,"statfs":{"total":96636764160,"available":90732535808,"internally_reserved":0,"allocated":4354883584,"data_stored":5217304961,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":426086,"internal_metadata":1548910490},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,6,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:04:55 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.47899999999999998,"15min":0.41499999999999998},"min":{"1min":0.184,"5min":0.17999999999999999,"15min":0.17000000000000001},"max":{"1min":3.282,"5min":4.101,"15min":4.1680000000000001},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.52100000000000002,"15min":0.497},"min":{"1min":0.20799999999999999,"5min":0.191,"15min":0.191},"max":{"1min":7.4880000000000004,"5min":7.4880000000000004,"15min":7.4880000000000004},"last":0.318}]},{"osd":2,"last update":"Tue Oct 17 02:05:05 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.59799999999999998,"15min":0.45700000000000002},"min":{"1min":0.23100000000000001,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":4.6470000000000002,"5min":6.0010000000000003,"15min":6.0010000000000003},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.59599999999999997,"15min":0.60899999999999999},"min":{"1min":0.25800000000000001,"5min":0.23100000000000001,"15min":0.19400000000000001},"max":{"1min":2.8959999999999999,"5min":8.0169999999999995,"15min":10.378},"last":0.28999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4353765376,"data_stored":5216362496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4222738432,"data_stored":5060571155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8572104704,"data_stored":10272534547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:05:30.053 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:05:30.053 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:05:41.492 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:41.487+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T02:05:41.533 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:41.527+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T02:05:41.534 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:41.527+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T02:05:41.534 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:41.527+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T02:05:41.534 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:41.527+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T02:05:49.498 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:49.482+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T02:05:49.499 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:49.490+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T02:05:49.512 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:49.494+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbda750 2023-10-17T02:05:49.512 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:49.494+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T02:05:49.513 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:49.494+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T02:05:49.513 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:49.494+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T02:05:49.513 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:49.494+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T02:05:49.514 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:49.494+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T02:05:49.514 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:49.494+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T02:05:49.514 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:49.494+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T02:05:49.515 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:49.494+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T02:05:49.515 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:49.494+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T02:05:49.515 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:49.494+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T02:05:49.515 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:49.494+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T02:05:49.516 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:05:49.494+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T02:05:50.055 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:05:50.813 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:05:50.814 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:05:50.862 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2619,"stamp":"2023-10-17T02:05:49.549825+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28253023064,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2127143,"num_read_kb":8308461,"num_write":2101965,"num_write_kb":8407511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87827,"ondisk_log_size":87827,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22001976,"kb_used_data":16807772,"kb_used_omap":1544,"kb_used_meta":5192631,"kb_avail":261113544,"statfs":{"total":289910292480,"available":267380269056,"internally_reserved":0,"allocated":17211158528,"data_stored":20610958038,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1581468,"internal_metadata":5317254756},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,8,14,13,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18931712,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4798,"num_read_kb":19124,"num_write":4644,"num_write_kb":18576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001912"},"pg_stats":[{"pgid":"2.7","version":"110'13905","reported_seq":19586,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:05:46.057447+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:05:46.057447+0000","last_peered":"2023-10-17T02:05:46.057447+0000","last_clean":"2023-10-17T02:05:46.057447+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:05:46.057447+0000","last_undegraded":"2023-10-17T02:05:46.057447+0000","last_fullsized":"2023-10-17T02:05:46.057447+0000","mapping_epoch":13,"log_start":"77'3900","ondisk_log_start":"77'3900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"110'10474","reported_seq":15909,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:05:46.055749+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:05:46.055749+0000","last_peered":"2023-10-17T02:05:46.055749+0000","last_clean":"2023-10-17T02:05:46.055749+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:05:46.055749+0000","last_undegraded":"2023-10-17T02:05:46.055749+0000","last_fullsized":"2023-10-17T02:05:46.055749+0000","mapping_epoch":13,"log_start":"44'400","ondisk_log_start":"44'400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"110'7022","reported_seq":9953,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:05:46.053441+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:05:46.053441+0000","last_peered":"2023-10-17T02:05:46.053441+0000","last_clean":"2023-10-17T02:05:46.053441+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:05:46.053441+0000","last_undegraded":"2023-10-17T02:05:46.053441+0000","last_fullsized":"2023-10-17T02:05:46.053441+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":7022,"log_dups_size":0,"ondisk_log_size":7022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"110'10509","reported_seq":15353,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:05:46.053294+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:05:46.053294+0000","last_peered":"2023-10-17T02:05:46.053294+0000","last_clean":"2023-10-17T02:05:46.053294+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:05:46.053294+0000","last_undegraded":"2023-10-17T02:05:46.053294+0000","last_fullsized":"2023-10-17T02:05:46.053294+0000","mapping_epoch":13,"log_start":"44'500","ondisk_log_start":"44'500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"110'535670","reported_seq":1067355,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:05:47.513198+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:05:47.513198+0000","last_peered":"2023-10-17T02:05:47.513198+0000","last_clean":"2023-10-17T02:05:47.513198+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:05:47.513198+0000","last_undegraded":"2023-10-17T02:05:47.513198+0000","last_fullsized":"2023-10-17T02:05:47.513198+0000","mapping_epoch":16,"log_start":"108'525610","ondisk_log_start":"108'525610","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7162798080,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":532372,"num_read_kb":2097092,"num_write":532913,"num_write_kb":2131652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":267,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:04:02.342251+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:04:02.342251+0000","last_peered":"2023-10-17T02:04:02.342251+0000","last_clean":"2023-10-17T02:04:02.342251+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:04:02.342251+0000","last_undegraded":"2023-10-17T02:04:02.342251+0000","last_fullsized":"2023-10-17T02:04:02.342251+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"110'514991","reported_seq":1027349,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:05:47.934668+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:05:47.934668+0000","last_peered":"2023-10-17T02:05:47.934668+0000","last_clean":"2023-10-17T02:05:47.934668+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:05:47.934668+0000","last_undegraded":"2023-10-17T02:05:47.934668+0000","last_fullsized":"2023-10-17T02:05:47.934668+0000","mapping_epoch":16,"log_start":"108'504919","ondisk_log_start":"108'504919","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6886133760,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":513050,"num_read_kb":2022900,"num_write":512341,"num_write_kb":2049364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2892,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:05:40.347151+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:05:40.347151+0000","last_peered":"2023-10-17T02:05:40.347151+0000","last_clean":"2023-10-17T02:05:40.347151+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:05:40.347151+0000","last_undegraded":"2023-10-17T02:05:40.347151+0000","last_fullsized":"2023-10-17T02:05:40.347151+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"110'536615","reported_seq":1070101,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:05:47.514671+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:05:47.514671+0000","last_peered":"2023-10-17T02:05:47.514671+0000","last_clean":"2023-10-17T02:05:47.514671+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:05:47.514671+0000","last_undegraded":"2023-10-17T02:05:47.514671+0000","last_fullsized":"2023-10-17T02:05:47.514671+0000","mapping_epoch":16,"log_start":"108'526525","ondisk_log_start":"108'526525","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7179251712,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":534210,"num_read_kb":2105164,"num_write":533861,"num_write_kb":2135444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4336,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:05:40.223478+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:05:40.223478+0000","last_peered":"2023-10-17T02:05:40.223478+0000","last_clean":"2023-10-17T02:05:40.223478+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:05:40.223478+0000","last_undegraded":"2023-10-17T02:05:40.223478+0000","last_fullsized":"2023-10-17T02:05:40.223478+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"110'523908","reported_seq":1046737,"reported_epoch":110,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:05:47.990433+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:05:47.990433+0000","last_peered":"2023-10-17T02:05:47.990433+0000","last_clean":"2023-10-17T02:05:47.990433+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:05:47.990433+0000","last_undegraded":"2023-10-17T02:05:47.990433+0000","last_fullsized":"2023-10-17T02:05:47.990433+0000","mapping_epoch":16,"log_start":"108'513818","ondisk_log_start":"108'513818","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":742,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 262s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7024246803,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":523549,"num_read_kb":2062652,"num_write":522220,"num_write_kb":2088874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"110'10478","reported_seq":14686,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:05:46.060418+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:05:46.060418+0000","last_peered":"2023-10-17T02:05:46.060418+0000","last_clean":"2023-10-17T02:05:46.060418+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:05:46.060418+0000","last_undegraded":"2023-10-17T02:05:46.060418+0000","last_fullsized":"2023-10-17T02:05:46.060418+0000","mapping_epoch":13,"log_start":"44'400","ondisk_log_start":"44'400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":288,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:04:42.362981+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:04:42.362981+0000","last_peered":"2023-10-17T02:04:42.362981+0000","last_clean":"2023-10-17T02:04:42.362981+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:04:42.362981+0000","last_undegraded":"2023-10-17T02:04:42.362981+0000","last_fullsized":"2023-10-17T02:04:42.362981+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28252430355,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2103181,"num_read_kb":8287808,"num_write":2101335,"num_write_kb":8405334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17209954304,"data_stored":20610813990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40312,"ondisk_log_size":40312,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":47416,"ondisk_log_size":47416,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739375,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10121060,"kb_used_data":8400796,"kb_used_omap":765,"kb_used_meta":1719490,"kb_avail":84250780,"statfs":{"total":96636764160,"available":86272798720,"internally_reserved":0,"allocated":8602415104,"data_stored":10302217371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":784319,"internal_metadata":1760757825},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:05:26 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.57299999999999995,"15min":0.45900000000000002},"min":{"1min":0.188,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":12.852,"5min":12.852,"15min":12.852},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.64100000000000001,"15min":0.54700000000000004},"min":{"1min":0.247,"5min":0.192,"15min":0.188},"max":{"1min":12.784000000000001,"5min":12.784000000000001,"15min":12.784000000000001},"last":0.32800000000000001}]},{"osd":1,"last update":"Tue Oct 17 02:05:00 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.439,"15min":0.48899999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.20699999999999999},"max":{"1min":3.7839999999999998,"5min":6.202,"15min":9.5429999999999993},"last":0.34399999999999997},{"interface":"front","average":{"1min":1.032,"5min":0.67000000000000004,"15min":0.623},"min":{"1min":0.27700000000000002,"5min":0.23200000000000001,"15min":0.19500000000000001},"max":{"1min":12.824999999999999,"5min":12.824999999999999,"15min":12.824999999999999},"last":0.309}]}]},{"osd":1,"up_from":8,"seq":34359739372,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6067992,"kb_used_data":4139020,"kb_used_omap":362,"kb_used_meta":1928597,"kb_avail":88303848,"statfs":{"total":96636764160,"available":90423140352,"internally_reserved":0,"allocated":4238356480,"data_stored":5075932346,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":371063,"internal_metadata":1974883977},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,3,3,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:05:21 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.54600000000000004,"15min":0.51000000000000001},"min":{"1min":0.17999999999999999,"5min":0.16600000000000001,"15min":0.16300000000000001},"max":{"1min":6.8049999999999997,"5min":6.8049999999999997,"15min":6.8049999999999997},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.59599999999999997,"15min":0.56000000000000005},"min":{"1min":0.24299999999999999,"5min":0.23000000000000001,"15min":0.22},"max":{"1min":6.8479999999999999,"5min":6.8479999999999999,"15min":6.8479999999999999},"last":0.318}]},{"osd":2,"last update":"Tue Oct 17 02:05:21 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.60599999999999998,"15min":0.53300000000000003},"min":{"1min":0.216,"5min":0.19500000000000001,"15min":0.188},"max":{"1min":0.92800000000000005,"5min":6.8230000000000004,"15min":6.8230000000000004},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.53900000000000003,"15min":0.499},"min":{"1min":0.249,"5min":0.22600000000000001,"15min":0.20100000000000001},"max":{"1min":0.89300000000000002,"5min":6.7800000000000002,"15min":6.7800000000000002},"last":0.375}]}]},{"osd":0,"up_from":8,"seq":34359739372,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5812924,"kb_used_data":4267956,"kb_used_omap":416,"kb_used_meta":1544543,"kb_avail":88558916,"statfs":{"total":96636764160,"available":90684329984,"internally_reserved":0,"allocated":4370386944,"data_stored":5232808321,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":426086,"internal_metadata":1581612954},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,7,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:04:55 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.47899999999999998,"15min":0.41499999999999998},"min":{"1min":0.184,"5min":0.17999999999999999,"15min":0.17000000000000001},"max":{"1min":3.282,"5min":4.101,"15min":4.1680000000000001},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.52100000000000002,"15min":0.497},"min":{"1min":0.20799999999999999,"5min":0.191,"15min":0.191},"max":{"1min":7.4880000000000004,"5min":7.4880000000000004,"15min":7.4880000000000004},"last":0.19800000000000001}]},{"osd":2,"last update":"Tue Oct 17 02:05:05 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.59799999999999998,"15min":0.45700000000000002},"min":{"1min":0.23100000000000001,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":4.6470000000000002,"5min":6.0010000000000003,"15min":6.0010000000000003},"last":0.245},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.59599999999999997,"15min":0.60899999999999999},"min":{"1min":0.25800000000000001,"5min":0.23100000000000001,"15min":0.19400000000000001},"max":{"1min":2.8959999999999999,"5min":8.0169999999999995,"15min":10.378},"last":0.29699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4369526784,"data_stored":5232123904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4237565952,"data_stored":5075398675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8602861568,"data_stored":10303291411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:05:50.865 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:05:50.865 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:06:10.867 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:06:11.308 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:06:11.308 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:06:11.341 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2629,"stamp":"2023-10-17T02:06:09.554754+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28283599704,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2134786,"num_read_kb":8338885,"num_write":2109439,"num_write_kb":8437407,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87641,"ondisk_log_size":87641,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21221684,"kb_used_data":16867160,"kb_used_omap":1546,"kb_used_meta":4352949,"kb_avail":261893836,"statfs":{"total":289910292480,"available":268179288064,"internally_reserved":0,"allocated":17271971840,"data_stored":20671771350,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1583147,"internal_metadata":4457420757},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,10,9,0,2,15,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23502848,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5746,"num_read_kb":22868,"num_write":5742,"num_write_kb":22968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003655"},"pg_stats":[{"pgid":"2.7","version":"110'13961","reported_seq":19658,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:06:07.332087+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:06:07.332087+0000","last_peered":"2023-10-17T02:06:07.332087+0000","last_clean":"2023-10-17T02:06:07.332087+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:06:07.332087+0000","last_undegraded":"2023-10-17T02:06:07.332087+0000","last_fullsized":"2023-10-17T02:06:07.332087+0000","mapping_epoch":13,"log_start":"77'3900","ondisk_log_start":"77'3900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"110'10516","reported_seq":15963,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:06:07.320996+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:06:07.320996+0000","last_peered":"2023-10-17T02:06:07.320996+0000","last_clean":"2023-10-17T02:06:07.320996+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:06:07.320996+0000","last_undegraded":"2023-10-17T02:06:07.320996+0000","last_fullsized":"2023-10-17T02:06:07.320996+0000","mapping_epoch":13,"log_start":"44'500","ondisk_log_start":"44'500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"110'7050","reported_seq":9990,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:06:07.302792+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:06:07.302792+0000","last_peered":"2023-10-17T02:06:07.302792+0000","last_clean":"2023-10-17T02:06:07.302792+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:06:07.302792+0000","last_undegraded":"2023-10-17T02:06:07.302792+0000","last_fullsized":"2023-10-17T02:06:07.302792+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":7050,"log_dups_size":0,"ondisk_log_size":7050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"110'10551","reported_seq":15408,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:06:07.325316+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:06:07.325316+0000","last_peered":"2023-10-17T02:06:07.325316+0000","last_clean":"2023-10-17T02:06:07.325316+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:06:07.325316+0000","last_undegraded":"2023-10-17T02:06:07.325316+0000","last_fullsized":"2023-10-17T02:06:07.325316+0000","mapping_epoch":13,"log_start":"44'500","ondisk_log_start":"44'500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"110'537552","reported_seq":1071211,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:06:07.691864+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:06:07.691864+0000","last_peered":"2023-10-17T02:06:07.691864+0000","last_clean":"2023-10-17T02:06:07.691864+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:06:07.691864+0000","last_undegraded":"2023-10-17T02:06:07.691864+0000","last_fullsized":"2023-10-17T02:06:07.691864+0000","mapping_epoch":16,"log_start":"109'527510","ondisk_log_start":"109'527510","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7170510848,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":534345,"num_read_kb":2104944,"num_write":534795,"num_write_kb":2139180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":268,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:05:22.369105+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:05:22.369105+0000","last_peered":"2023-10-17T02:05:22.369105+0000","last_clean":"2023-10-17T02:05:22.369105+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:05:22.369105+0000","last_undegraded":"2023-10-17T02:05:22.369105+0000","last_fullsized":"2023-10-17T02:05:22.369105+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"110'516850","reported_seq":1031167,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:06:08.081398+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:06:08.081398+0000","last_peered":"2023-10-17T02:06:08.081398+0000","last_clean":"2023-10-17T02:06:08.081398+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:06:08.081398+0000","last_undegraded":"2023-10-17T02:06:08.081398+0000","last_fullsized":"2023-10-17T02:06:08.081398+0000","mapping_epoch":16,"log_start":"109'506819","ondisk_log_start":"109'506819","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6893727744,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":514999,"num_read_kb":2030668,"num_write":514200,"num_write_kb":2056800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2901,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:06:00.768414+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:06:00.768414+0000","last_peered":"2023-10-17T02:06:00.768414+0000","last_clean":"2023-10-17T02:06:00.768414+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:06:00.768414+0000","last_undegraded":"2023-10-17T02:06:00.768414+0000","last_fullsized":"2023-10-17T02:06:00.768414+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"110'538529","reported_seq":1073916,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:06:07.698441+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:06:07.698441+0000","last_peered":"2023-10-17T02:06:07.698441+0000","last_clean":"2023-10-17T02:06:07.698441+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:06:07.698441+0000","last_undegraded":"2023-10-17T02:06:07.698441+0000","last_fullsized":"2023-10-17T02:06:07.698441+0000","mapping_epoch":16,"log_start":"109'528525","ondisk_log_start":"109'528525","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7187079168,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":536107,"num_read_kb":2112712,"num_write":535775,"num_write_kb":2143100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4348,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:06:00.708229+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:06:00.708229+0000","last_peered":"2023-10-17T02:06:00.708229+0000","last_clean":"2023-10-17T02:06:00.708229+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:06:00.708229+0000","last_undegraded":"2023-10-17T02:06:00.708229+0000","last_fullsized":"2023-10-17T02:06:00.708229+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"110'525757","reported_seq":1050418,"reported_epoch":110,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:06:08.092555+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:06:08.092555+0000","last_peered":"2023-10-17T02:06:08.092555+0000","last_clean":"2023-10-17T02:06:08.092555+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:06:08.092555+0000","last_undegraded":"2023-10-17T02:06:08.092555+0000","last_fullsized":"2023-10-17T02:06:08.092555+0000","mapping_epoch":16,"log_start":"109'515718","ondisk_log_start":"109'515718","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":802,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 282s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7031689235,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":525373,"num_read_kb":2069908,"num_write":524039,"num_write_kb":2096150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"110'10520","reported_seq":14741,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:06:07.345537+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:06:07.345537+0000","last_peered":"2023-10-17T02:06:07.345537+0000","last_clean":"2023-10-17T02:06:07.345537+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:06:07.345537+0000","last_undegraded":"2023-10-17T02:06:07.345537+0000","last_fullsized":"2023-10-17T02:06:07.345537+0000","mapping_epoch":13,"log_start":"46'500","ondisk_log_start":"46'500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":288,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:04:42.362981+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:04:42.362981+0000","last_peered":"2023-10-17T02:04:42.362981+0000","last_clean":"2023-10-17T02:04:42.362981+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:04:42.362981+0000","last_undegraded":"2023-10-17T02:04:42.362981+0000","last_fullsized":"2023-10-17T02:04:42.362981+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28283006995,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2110824,"num_read_kb":8318232,"num_write":2108809,"num_write_kb":8435230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17270964224,"data_stored":20671823910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40116,"ondisk_log_size":40116,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":47426,"ondisk_log_size":47426,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739379,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10253460,"kb_used_data":8430536,"kb_used_omap":765,"kb_used_meta":1822146,"kb_avail":84118380,"statfs":{"total":96636764160,"available":86137221120,"internally_reserved":0,"allocated":8632868864,"data_stored":10332671131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":784319,"internal_metadata":1865877569},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:05:26 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.57299999999999995,"15min":0.45900000000000002},"min":{"1min":0.188,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":12.852,"5min":12.852,"15min":12.852},"last":2.8479999999999999},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.64100000000000001,"15min":0.54700000000000004},"min":{"1min":0.247,"5min":0.192,"15min":0.188},"max":{"1min":12.784000000000001,"5min":12.784000000000001,"15min":12.784000000000001},"last":2.8090000000000002}]},{"osd":1,"last update":"Tue Oct 17 02:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.48599999999999999,"15min":0.496},"min":{"1min":0.217,"5min":0.21199999999999999,"15min":0.20699999999999999},"max":{"1min":3.9020000000000001,"5min":6.202,"15min":9.5429999999999993},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.71699999999999997,"15min":0.63100000000000001},"min":{"1min":0.26500000000000001,"5min":0.23200000000000001,"15min":0.23100000000000001},"max":{"1min":3.9350000000000001,"5min":12.824999999999999,"15min":12.824999999999999},"last":0.35799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739376,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5086116,"kb_used_data":4153628,"kb_used_omap":364,"kb_used_meta":932115,"kb_avail":89285724,"statfs":{"total":96636764160,"available":91428581376,"internally_reserved":0,"allocated":4253315072,"data_stored":5090890938,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":372742,"internal_metadata":954486778},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,8,0,0,11,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:05:21 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.54600000000000004,"15min":0.51000000000000001},"min":{"1min":0.17999999999999999,"5min":0.16600000000000001,"15min":0.16300000000000001},"max":{"1min":6.8049999999999997,"5min":6.8049999999999997,"15min":6.8049999999999997},"last":0.317},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.59599999999999997,"15min":0.56000000000000005},"min":{"1min":0.24299999999999999,"5min":0.23000000000000001,"15min":0.22},"max":{"1min":6.8479999999999999,"5min":6.8479999999999999,"15min":6.8479999999999999},"last":0.35699999999999998}]},{"osd":2,"last update":"Tue Oct 17 02:05:21 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.60599999999999998,"15min":0.53300000000000003},"min":{"1min":0.216,"5min":0.19500000000000001,"15min":0.188},"max":{"1min":0.92800000000000005,"5min":6.8230000000000004,"15min":6.8230000000000004},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.53900000000000003,"15min":0.499},"min":{"1min":0.249,"5min":0.22600000000000001,"15min":0.20100000000000001},"max":{"1min":0.89300000000000002,"5min":6.7800000000000002,"15min":6.7800000000000002},"last":0.248}]}]},{"osd":0,"up_from":8,"seq":34359739376,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5882108,"kb_used_data":4282996,"kb_used_omap":416,"kb_used_meta":1598687,"kb_avail":88489732,"statfs":{"total":96636764160,"available":90613485568,"internally_reserved":0,"allocated":4385787904,"data_stored":5248209281,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":426086,"internal_metadata":1637056410},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,3,1,0,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:05:56 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.47599999999999998,"15min":0.433},"min":{"1min":0.20499999999999999,"5min":0.17999999999999999,"15min":0.17000000000000001},"max":{"1min":6.1360000000000001,"5min":6.1360000000000001,"15min":6.1360000000000001},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.59599999999999997,"15min":0.495},"min":{"1min":0.19800000000000001,"5min":0.191,"15min":0.191},"max":{"1min":6.5910000000000002,"5min":7.4880000000000004,"15min":7.4880000000000004},"last":0.46400000000000002}]},{"osd":2,"last update":"Tue Oct 17 02:06:06 2023","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":0.73599999999999999,"15min":0.52400000000000002},"min":{"1min":0.224,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":10.125,"5min":10.125,"15min":10.125},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.58499999999999996,"15min":0.623},"min":{"1min":0.251,"5min":0.23699999999999999,"15min":0.19400000000000001},"max":{"1min":6.2329999999999997,"5min":6.3899999999999997,"15min":10.378},"last":0.48899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4385054720,"data_stored":5247651840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4252626944,"data_stored":5090459667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8633282560,"data_stored":10333712403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:06:11.344 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:06:11.344 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:06:31.345 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:06:32.335 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:06:32.336 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:06:32.365 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2640,"stamp":"2023-10-17T02:06:31.558843+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28314504024,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2142569,"num_read_kb":8369857,"num_write":2117011,"num_write_kb":8467695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87876,"ondisk_log_size":87876,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21602764,"kb_used_data":16936624,"kb_used_omap":1546,"kb_used_meta":4664565,"kb_avail":261512756,"statfs":{"total":289910292480,"available":267789062144,"internally_reserved":0,"allocated":17343102976,"data_stored":20742902486,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1583147,"internal_metadata":4776515541},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,7,5,8,8,9,9],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15372288,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3872,"num_read_kb":15420,"num_write":3764,"num_write_kb":15056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002335"},"pg_stats":[{"pgid":"2.7","version":"110'13997","reported_seq":19710,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:06:27.671382+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:06:27.671382+0000","last_peered":"2023-10-17T02:06:27.671382+0000","last_clean":"2023-10-17T02:06:27.671382+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:06:27.671382+0000","last_undegraded":"2023-10-17T02:06:27.671382+0000","last_fullsized":"2023-10-17T02:06:27.671382+0000","mapping_epoch":13,"log_start":"77'3900","ondisk_log_start":"77'3900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"110'10543","reported_seq":16002,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:06:27.661357+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:06:27.661357+0000","last_peered":"2023-10-17T02:06:27.661357+0000","last_clean":"2023-10-17T02:06:27.661357+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:06:27.661357+0000","last_undegraded":"2023-10-17T02:06:27.661357+0000","last_fullsized":"2023-10-17T02:06:27.661357+0000","mapping_epoch":13,"log_start":"44'500","ondisk_log_start":"44'500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"110'7068","reported_seq":10016,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:06:27.650677+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:06:27.650677+0000","last_peered":"2023-10-17T02:06:27.650677+0000","last_clean":"2023-10-17T02:06:27.650677+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:06:27.650677+0000","last_undegraded":"2023-10-17T02:06:27.650677+0000","last_fullsized":"2023-10-17T02:06:27.650677+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":7068,"log_dups_size":0,"ondisk_log_size":7068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"110'10578","reported_seq":15447,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:06:27.667432+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:06:27.667432+0000","last_peered":"2023-10-17T02:06:27.667432+0000","last_clean":"2023-10-17T02:06:27.667432+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:06:27.667432+0000","last_undegraded":"2023-10-17T02:06:27.667432+0000","last_fullsized":"2023-10-17T02:06:27.667432+0000","mapping_epoch":13,"log_start":"44'500","ondisk_log_start":"44'500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"110'539538","reported_seq":1075152,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:06:27.765333+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:06:27.765333+0000","last_peered":"2023-10-17T02:06:27.765333+0000","last_clean":"2023-10-17T02:06:27.765333+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:06:27.765333+0000","last_undegraded":"2023-10-17T02:06:27.765333+0000","last_fullsized":"2023-10-17T02:06:27.765333+0000","mapping_epoch":16,"log_start":"109'529510","ondisk_log_start":"109'529510","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7178592256,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":536304,"num_read_kb":2112724,"num_write":536781,"num_write_kb":2147124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":268,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:05:22.369105+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:05:22.369105+0000","last_peered":"2023-10-17T02:05:22.369105+0000","last_clean":"2023-10-17T02:05:22.369105+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:05:22.369105+0000","last_undegraded":"2023-10-17T02:05:22.369105+0000","last_fullsized":"2023-10-17T02:05:22.369105+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"110'518700","reported_seq":1034900,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:06:28.231077+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:06:28.231077+0000","last_peered":"2023-10-17T02:06:28.231077+0000","last_clean":"2023-10-17T02:06:28.231077+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:06:28.231077+0000","last_undegraded":"2023-10-17T02:06:28.231077+0000","last_fullsized":"2023-10-17T02:06:28.231077+0000","mapping_epoch":16,"log_start":"109'508619","ondisk_log_start":"109'508619","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6901256192,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":516888,"num_read_kb":2038180,"num_write":516050,"num_write_kb":2064200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2911,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:06:26.303353+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:06:26.303353+0000","last_peered":"2023-10-17T02:06:26.303353+0000","last_clean":"2023-10-17T02:06:26.303353+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:06:26.303353+0000","last_undegraded":"2023-10-17T02:06:26.303353+0000","last_fullsized":"2023-10-17T02:06:26.303353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"110'540387","reported_seq":1077758,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:06:27.778809+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:06:27.778809+0000","last_peered":"2023-10-17T02:06:27.778809+0000","last_clean":"2023-10-17T02:06:27.778809+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:06:27.778809+0000","last_undegraded":"2023-10-17T02:06:27.778809+0000","last_fullsized":"2023-10-17T02:06:27.778809+0000","mapping_epoch":16,"log_start":"109'530325","ondisk_log_start":"109'530325","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7194685440,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":538095,"num_read_kb":2120636,"num_write":537633,"num_write_kb":2150532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4363,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:06:26.303661+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:06:26.303661+0000","last_peered":"2023-10-17T02:06:26.303661+0000","last_clean":"2023-10-17T02:06:26.303661+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:06:26.303661+0000","last_undegraded":"2023-10-17T02:06:26.303661+0000","last_fullsized":"2023-10-17T02:06:26.303661+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"110'527663","reported_seq":1054273,"reported_epoch":110,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:06:28.243951+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:06:28.243951+0000","last_peered":"2023-10-17T02:06:28.243951+0000","last_clean":"2023-10-17T02:06:28.243951+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:06:28.243951+0000","last_undegraded":"2023-10-17T02:06:28.243951+0000","last_fullsized":"2023-10-17T02:06:28.243951+0000","mapping_epoch":16,"log_start":"109'517618","ondisk_log_start":"109'517618","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":852,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 302s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7039377427,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":527320,"num_read_kb":2077664,"num_write":525917,"num_write_kb":2103662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"110'10547","reported_seq":14780,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:06:27.698254+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:06:27.698254+0000","last_peered":"2023-10-17T02:06:27.698254+0000","last_clean":"2023-10-17T02:06:27.698254+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:06:27.698254+0000","last_undegraded":"2023-10-17T02:06:27.698254+0000","last_fullsized":"2023-10-17T02:06:27.698254+0000","mapping_epoch":13,"log_start":"46'500","ondisk_log_start":"46'500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":288,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-17T02:04:42.362981+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:04:42.362981+0000","last_peered":"2023-10-17T02:04:42.362981+0000","last_clean":"2023-10-17T02:04:42.362981+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:04:42.362981+0000","last_undegraded":"2023-10-17T02:04:42.362981+0000","last_fullsized":"2023-10-17T02:04:42.362981+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28313911315,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2118607,"num_read_kb":8349204,"num_write":2116381,"num_write_kb":8465518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17340850176,"data_stored":20741709862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40216,"ondisk_log_size":40216,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":47561,"ondisk_log_size":47561,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739384,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10452440,"kb_used_data":8469712,"kb_used_omap":765,"kb_used_meta":1981954,"kb_avail":83919400,"statfs":{"total":96636764160,"available":85933465600,"internally_reserved":0,"allocated":8672985088,"data_stored":10372787355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":784319,"internal_metadata":2029520961},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:06:27 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.57699999999999996,"15min":0.46400000000000002},"min":{"1min":0.189,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":2.8479999999999999,"5min":12.852,"15min":12.852},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.439,"5min":0.64100000000000001,"15min":0.54400000000000004},"min":{"1min":0.20499999999999999,"5min":0.192,"15min":0.188},"max":{"1min":2.8090000000000002,"5min":12.784000000000001,"15min":12.784000000000001},"last":0.32000000000000001}]},{"osd":1,"last update":"Tue Oct 17 02:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.48599999999999999,"15min":0.496},"min":{"1min":0.217,"5min":0.21199999999999999,"15min":0.20699999999999999},"max":{"1min":3.9020000000000001,"5min":6.202,"15min":9.5429999999999993},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.71699999999999997,"15min":0.63100000000000001},"min":{"1min":0.26500000000000001,"5min":0.23200000000000001,"15min":0.23100000000000001},"max":{"1min":3.9350000000000001,"5min":12.824999999999999,"15min":12.824999999999999},"last":0.38900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739380,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5180768,"kb_used_data":4168664,"kb_used_omap":364,"kb_used_meta":1011731,"kb_avail":89191072,"statfs":{"total":96636764160,"available":91331657728,"internally_reserved":0,"allocated":4268711936,"data_stored":5106287802,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":372742,"internal_metadata":1036013562},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,2,6,7,9,9],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:06:24 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.54700000000000004,"15min":0.49299999999999999},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.16300000000000001},"max":{"1min":1.5820000000000001,"5min":6.8049999999999997,"15min":6.8049999999999997},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.59499999999999997,"15min":0.54300000000000004},"min":{"1min":0.249,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":1.613,"5min":6.8479999999999999,"15min":6.8479999999999999},"last":0.37}]},{"osd":2,"last update":"Tue Oct 17 02:06:24 2023","interfaces":[{"interface":"back","average":{"1min":1.9239999999999999,"5min":0.81399999999999995,"15min":0.61399999999999999},"min":{"1min":0.23499999999999999,"5min":0.19500000000000001,"15min":0.188},"max":{"1min":25.128,"5min":25.128,"15min":25.128},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.45000000000000001,"15min":0.47499999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.20100000000000001},"max":{"1min":1.4219999999999999,"5min":5.2089999999999996,"15min":6.7800000000000002},"last":0.308}]}]},{"osd":0,"up_from":8,"seq":34359739380,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5969556,"kb_used_data":4298248,"kb_used_omap":416,"kb_used_meta":1670879,"kb_avail":88402284,"statfs":{"total":96636764160,"available":90523938816,"internally_reserved":0,"allocated":4401405952,"data_stored":5263827329,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":426086,"internal_metadata":1710981018},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:05:56 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.47599999999999998,"15min":0.433},"min":{"1min":0.20499999999999999,"5min":0.17999999999999999,"15min":0.17000000000000001},"max":{"1min":6.1360000000000001,"5min":6.1360000000000001,"15min":6.1360000000000001},"last":5.1130000000000004},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.59599999999999997,"15min":0.495},"min":{"1min":0.19800000000000001,"5min":0.191,"15min":0.191},"max":{"1min":6.5910000000000002,"5min":7.4880000000000004,"15min":7.4880000000000004},"last":0.253}]},{"osd":2,"last update":"Tue Oct 17 02:06:06 2023","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":0.73599999999999999,"15min":0.52400000000000002},"min":{"1min":0.224,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":10.125,"5min":10.125,"15min":10.125},"last":0.318},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.58499999999999996,"15min":0.623},"min":{"1min":0.251,"5min":0.23699999999999999,"15min":0.19400000000000001},"max":{"1min":6.2329999999999997,"5min":6.3899999999999997,"15min":10.378},"last":0.35699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4400799744,"data_stored":5263396864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4267892736,"data_stored":5105725459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8672157696,"data_stored":10372587539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:06:32.368 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:06:32.368 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:06:33.327 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:06:33.321+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T02:06:33.328 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:06:33.321+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T02:06:33.328 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:06:33.321+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbda750 2023-10-17T02:06:33.328 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:06:33.321+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T02:06:33.328 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:06:33.321+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T02:06:33.329 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:06:33.321+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T02:06:33.329 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:06:33.321+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T02:06:33.329 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:06:33.321+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T02:06:33.330 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:06:33.321+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T02:06:33.330 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:06:33.321+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T02:06:33.331 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:06:33.321+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T02:06:33.331 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:06:33.321+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T02:06:33.331 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:06:33.321+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T02:06:33.332 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:06:33.321+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T02:06:33.332 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:06:33.321+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T02:06:37.712 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:06:37.709+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T02:06:37.726 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:06:37.721+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T02:06:37.727 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:06:37.721+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T02:06:37.730 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:06:37.725+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T02:06:37.730 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:06:37.725+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T02:06:52.369 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:06:52.963 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:06:52.964 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:06:52.994 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2651,"stamp":"2023-10-17T02:06:51.563913+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28344740696,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2150594,"num_read_kb":8401701,"num_write":2124427,"num_write_kb":8497359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88045,"ondisk_log_size":88045,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20898896,"kb_used_data":16995828,"kb_used_omap":1613,"kb_used_meta":3901426,"kb_avail":262216624,"statfs":{"total":289910292480,"available":268509822976,"internally_reserved":0,"allocated":17403727872,"data_stored":20803513921,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1651805,"internal_metadata":3995061155},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,4,3,10,10,6,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15355904,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3995,"num_read_kb":15856,"num_write":3753,"num_write_kb":15012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003732"},"pg_stats":[{"pgid":"2.7","version":"111'14058","reported_seq":19788,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:06:46.783819+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:06:46.783819+0000","last_peered":"2023-10-17T02:06:46.783819+0000","last_clean":"2023-10-17T02:06:46.783819+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:06:46.783819+0000","last_undegraded":"2023-10-17T02:06:46.783819+0000","last_fullsized":"2023-10-17T02:06:46.783819+0000","mapping_epoch":13,"log_start":"79'4000","ondisk_log_start":"79'4000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"111'10589","reported_seq":16061,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:06:46.783516+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:06:46.783516+0000","last_peered":"2023-10-17T02:06:46.783516+0000","last_clean":"2023-10-17T02:06:46.783516+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:06:46.783516+0000","last_undegraded":"2023-10-17T02:06:46.783516+0000","last_fullsized":"2023-10-17T02:06:46.783516+0000","mapping_epoch":13,"log_start":"44'500","ondisk_log_start":"44'500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"111'7099","reported_seq":10056,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:06:46.783422+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:06:46.783422+0000","last_peered":"2023-10-17T02:06:46.783422+0000","last_clean":"2023-10-17T02:06:46.783422+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:06:46.783422+0000","last_undegraded":"2023-10-17T02:06:46.783422+0000","last_fullsized":"2023-10-17T02:06:46.783422+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":7099,"log_dups_size":0,"ondisk_log_size":7099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"111'10624","reported_seq":15506,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:06:46.783628+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:06:46.783628+0000","last_peered":"2023-10-17T02:06:46.783628+0000","last_clean":"2023-10-17T02:06:46.783628+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:06:46.783628+0000","last_undegraded":"2023-10-17T02:06:46.783628+0000","last_fullsized":"2023-10-17T02:06:46.783628+0000","mapping_epoch":13,"log_start":"46'600","ondisk_log_start":"46'600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"111'541389","reported_seq":1079043,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:06:47.863568+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:06:47.863568+0000","last_peered":"2023-10-17T02:06:47.863568+0000","last_clean":"2023-10-17T02:06:47.863568+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:06:47.863568+0000","last_undegraded":"2023-10-17T02:06:47.863568+0000","last_fullsized":"2023-10-17T02:06:47.863568+0000","mapping_epoch":16,"log_start":"109'531310","ondisk_log_start":"109'531310","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7186149376,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":538342,"num_read_kb":2120792,"num_write":538632,"num_write_kb":2154528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":269,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:06:02.679940+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:06:02.679940+0000","last_peered":"2023-10-17T02:06:02.679940+0000","last_clean":"2023-10-17T02:06:02.679940+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:06:02.679940+0000","last_undegraded":"2023-10-17T02:06:02.679940+0000","last_fullsized":"2023-10-17T02:06:02.679940+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"111'520506","reported_seq":1038688,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:06:48.400125+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:06:48.400125+0000","last_peered":"2023-10-17T02:06:48.400125+0000","last_clean":"2023-10-17T02:06:48.400125+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:06:48.400125+0000","last_undegraded":"2023-10-17T02:06:48.400125+0000","last_fullsized":"2023-10-17T02:06:48.400125+0000","mapping_epoch":16,"log_start":"109'510419","ondisk_log_start":"109'510419","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6908608512,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":518864,"num_read_kb":2046024,"num_write":517856,"num_write_kb":2071424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2920,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:06:46.784024+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:06:46.784024+0000","last_peered":"2023-10-17T02:06:46.784024+0000","last_clean":"2023-10-17T02:06:46.784024+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:06:46.784024+0000","last_undegraded":"2023-10-17T02:06:46.784024+0000","last_fullsized":"2023-10-17T02:06:46.784024+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"111'542321","reported_seq":1081745,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:06:47.864679+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:06:47.864679+0000","last_peered":"2023-10-17T02:06:47.864679+0000","last_clean":"2023-10-17T02:06:47.864679+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:06:47.864679+0000","last_undegraded":"2023-10-17T02:06:47.864679+0000","last_fullsized":"2023-10-17T02:06:47.864679+0000","mapping_epoch":16,"log_start":"109'532225","ondisk_log_start":"109'532225","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7202582528,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":540141,"num_read_kb":2128764,"num_write":539567,"num_write_kb":2158268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4377,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:06:46.787537+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:06:46.787537+0000","last_peered":"2023-10-17T02:06:46.787537+0000","last_clean":"2023-10-17T02:06:46.787537+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:06:46.787537+0000","last_undegraded":"2023-10-17T02:06:46.787537+0000","last_fullsized":"2023-10-17T02:06:46.787537+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"111'529511","reported_seq":1058099,"reported_epoch":111,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:06:48.404588+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:06:48.404588+0000","last_peered":"2023-10-17T02:06:48.404588+0000","last_clean":"2023-10-17T02:06:48.404588+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:06:48.404588+0000","last_undegraded":"2023-10-17T02:06:48.404588+0000","last_fullsized":"2023-10-17T02:06:48.404588+0000","mapping_epoch":16,"log_start":"109'519418","ondisk_log_start":"109'519418","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":892,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 322s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7046807571,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":529285,"num_read_kb":2085468,"num_write":527742,"num_write_kb":2110962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"111'10593","reported_seq":14839,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:06:46.783358+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:06:46.783358+0000","last_peered":"2023-10-17T02:06:46.783358+0000","last_clean":"2023-10-17T02:06:46.783358+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:06:46.783358+0000","last_undegraded":"2023-10-17T02:06:46.783358+0000","last_fullsized":"2023-10-17T02:06:46.783358+0000","mapping_epoch":13,"log_start":"46'500","ondisk_log_start":"46'500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":290,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:06:32.401237+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:06:32.401237+0000","last_peered":"2023-10-17T02:06:32.401237+0000","last_clean":"2023-10-17T02:06:32.401237+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:06:32.401237+0000","last_undegraded":"2023-10-17T02:06:32.401237+0000","last_fullsized":"2023-10-17T02:06:32.401237+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28344147987,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2126632,"num_read_kb":8381048,"num_write":2123797,"num_write_kb":8495182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17401466880,"data_stored":20802326566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40355,"ondisk_log_size":40355,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":47591,"ondisk_log_size":47591,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739388,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9648060,"kb_used_data":8499184,"kb_used_omap":832,"kb_used_meta":1148031,"kb_avail":84723780,"statfs":{"total":96636764160,"available":86757150720,"internally_reserved":0,"allocated":8703164416,"data_stored":10402962196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":852977,"internal_metadata":1175583759},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:06:27 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.57699999999999996,"15min":0.46400000000000002},"min":{"1min":0.189,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":2.8479999999999999,"5min":12.852,"15min":12.852},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.439,"5min":0.64100000000000001,"15min":0.54400000000000004},"min":{"1min":0.20499999999999999,"5min":0.192,"15min":0.188},"max":{"1min":2.8090000000000002,"5min":12.784000000000001,"15min":12.784000000000001},"last":0.41499999999999998}]},{"osd":1,"last update":"Tue Oct 17 02:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.48599999999999999,"15min":0.496},"min":{"1min":0.217,"5min":0.21199999999999999,"15min":0.20699999999999999},"max":{"1min":3.9020000000000001,"5min":6.202,"15min":9.5429999999999993},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.71699999999999997,"15min":0.63100000000000001},"min":{"1min":0.26500000000000001,"5min":0.23200000000000001,"15min":0.23100000000000001},"max":{"1min":3.9350000000000001,"5min":12.824999999999999,"15min":12.824999999999999},"last":0.39700000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739384,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5230060,"kb_used_data":4183268,"kb_used_omap":364,"kb_used_meta":1046419,"kb_avail":89141780,"statfs":{"total":96636764160,"available":91281182720,"internally_reserved":0,"allocated":4283666432,"data_stored":5121237811,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":372742,"internal_metadata":1071534074},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,6,6,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:06:24 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.54700000000000004,"15min":0.49299999999999999},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.16300000000000001},"max":{"1min":1.5820000000000001,"5min":6.8049999999999997,"15min":6.8049999999999997},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.59499999999999997,"15min":0.54300000000000004},"min":{"1min":0.249,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":1.613,"5min":6.8479999999999999,"15min":6.8479999999999999},"last":0.33500000000000002}]},{"osd":2,"last update":"Tue Oct 17 02:06:24 2023","interfaces":[{"interface":"back","average":{"1min":1.9239999999999999,"5min":0.81399999999999995,"15min":0.61399999999999999},"min":{"1min":0.23499999999999999,"5min":0.19500000000000001,"15min":0.188},"max":{"1min":25.128,"5min":25.128,"15min":25.128},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.45000000000000001,"15min":0.47499999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.20100000000000001},"max":{"1min":1.4219999999999999,"5min":5.2089999999999996,"15min":6.7800000000000002},"last":0.28299999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739384,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6020776,"kb_used_data":4313376,"kb_used_omap":416,"kb_used_meta":1706975,"kb_avail":88351064,"statfs":{"total":96636764160,"available":90471489536,"internally_reserved":0,"allocated":4416897024,"data_stored":5279313914,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":426086,"internal_metadata":1747943322},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,5,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:05:56 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.47599999999999998,"15min":0.433},"min":{"1min":0.20499999999999999,"5min":0.17999999999999999,"15min":0.17000000000000001},"max":{"1min":6.1360000000000001,"5min":6.1360000000000001,"15min":6.1360000000000001},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.59599999999999997,"15min":0.495},"min":{"1min":0.19800000000000001,"5min":0.191,"15min":0.191},"max":{"1min":6.5910000000000002,"5min":7.4880000000000004,"15min":7.4880000000000004},"last":0.28499999999999998}]},{"osd":2,"last update":"Tue Oct 17 02:06:06 2023","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":0.73599999999999999,"15min":0.52400000000000002},"min":{"1min":0.224,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":10.125,"5min":10.125,"15min":10.125},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.58499999999999996,"15min":0.623},"min":{"1min":0.251,"5min":0.23699999999999999,"15min":0.19400000000000001},"max":{"1min":6.2329999999999997,"5min":6.3899999999999997,"15min":10.378},"last":0.246}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4416307200,"data_stored":5278904320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4282765312,"data_stored":5120598035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8702394368,"data_stored":10402824211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:06:52.996 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:06:52.997 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:07:12.998 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:07:13.599 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:07:13.599 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:07:13.629 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2662,"stamp":"2023-10-17T02:07:13.567699+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28383189848,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2160613,"num_read_kb":8441449,"num_write":2133853,"num_write_kb":8535063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87762,"ondisk_log_size":87762,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21177120,"kb_used_data":17064136,"kb_used_omap":1613,"kb_used_meta":4111346,"kb_avail":261938400,"statfs":{"total":289910292480,"available":268224921600,"internally_reserved":0,"allocated":17473675264,"data_stored":20873461313,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1651805,"internal_metadata":4210019235},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,5,9,7,15,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23273472,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5843,"num_read_kb":23208,"num_write":5693,"num_write_kb":22772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002055"},"pg_stats":[{"pgid":"2.7","version":"111'14102","reported_seq":19853,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:07:12.383861+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:07:12.383861+0000","last_peered":"2023-10-17T02:07:12.383861+0000","last_clean":"2023-10-17T02:07:12.383861+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:07:12.383861+0000","last_undegraded":"2023-10-17T02:07:12.383861+0000","last_fullsized":"2023-10-17T02:07:12.383861+0000","mapping_epoch":13,"log_start":"79'4100","ondisk_log_start":"79'4100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"111'10622","reported_seq":16110,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:07:12.383410+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:07:12.383410+0000","last_peered":"2023-10-17T02:07:12.383410+0000","last_clean":"2023-10-17T02:07:12.383410+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:07:12.383410+0000","last_undegraded":"2023-10-17T02:07:12.383410+0000","last_fullsized":"2023-10-17T02:07:12.383410+0000","mapping_epoch":13,"log_start":"46'600","ondisk_log_start":"46'600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"111'7121","reported_seq":10088,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:07:12.383189+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:07:12.383189+0000","last_peered":"2023-10-17T02:07:12.383189+0000","last_clean":"2023-10-17T02:07:12.383189+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:07:12.383189+0000","last_undegraded":"2023-10-17T02:07:12.383189+0000","last_fullsized":"2023-10-17T02:07:12.383189+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":7121,"log_dups_size":0,"ondisk_log_size":7121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"111'10657","reported_seq":15554,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:07:12.383680+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:07:12.383680+0000","last_peered":"2023-10-17T02:07:12.383680+0000","last_clean":"2023-10-17T02:07:12.383680+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:07:12.383680+0000","last_undegraded":"2023-10-17T02:07:12.383680+0000","last_fullsized":"2023-10-17T02:07:12.383680+0000","mapping_epoch":13,"log_start":"46'600","ondisk_log_start":"46'600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"111'543789","reported_seq":1084016,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:07:12.990976+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:07:12.990976+0000","last_peered":"2023-10-17T02:07:12.990976+0000","last_clean":"2023-10-17T02:07:12.990976+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:07:12.990976+0000","last_undegraded":"2023-10-17T02:07:12.990976+0000","last_fullsized":"2023-10-17T02:07:12.990976+0000","mapping_epoch":16,"log_start":"110'533710","ondisk_log_start":"110'533710","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7195967488,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":540911,"num_read_kb":2130968,"num_write":541032,"num_write_kb":2164128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":269,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:06:02.679940+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:06:02.679940+0000","last_peered":"2023-10-17T02:06:02.679940+0000","last_clean":"2023-10-17T02:06:02.679940+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:06:02.679940+0000","last_undegraded":"2023-10-17T02:06:02.679940+0000","last_fullsized":"2023-10-17T02:06:02.679940+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"111'522823","reported_seq":1043438,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:07:13.524072+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:07:13.524072+0000","last_peered":"2023-10-17T02:07:13.524072+0000","last_clean":"2023-10-17T02:07:13.524072+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:07:13.524072+0000","last_undegraded":"2023-10-17T02:07:13.524072+0000","last_fullsized":"2023-10-17T02:07:13.524072+0000","mapping_epoch":16,"log_start":"110'512819","ondisk_log_start":"110'512819","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6918057984,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":521298,"num_read_kb":2055684,"num_write":520173,"num_write_kb":2080692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2928,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:07:07.372728+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:07:07.372728+0000","last_peered":"2023-10-17T02:07:07.372728+0000","last_clean":"2023-10-17T02:07:07.372728+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:07:07.372728+0000","last_undegraded":"2023-10-17T02:07:07.372728+0000","last_fullsized":"2023-10-17T02:07:07.372728+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"111'544707","reported_seq":1086678,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:07:12.971686+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:07:12.971686+0000","last_peered":"2023-10-17T02:07:12.971686+0000","last_clean":"2023-10-17T02:07:12.971686+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:07:12.971686+0000","last_undegraded":"2023-10-17T02:07:12.971686+0000","last_fullsized":"2023-10-17T02:07:12.971686+0000","mapping_epoch":16,"log_start":"110'534625","ondisk_log_start":"110'534625","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7212281856,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":542697,"num_read_kb":2138904,"num_write":541953,"num_write_kb":2167812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4389,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:07:07.264228+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:07:07.264228+0000","last_peered":"2023-10-17T02:07:07.264228+0000","last_clean":"2023-10-17T02:07:07.264228+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:07:07.264228+0000","last_undegraded":"2023-10-17T02:07:07.264228+0000","last_fullsized":"2023-10-17T02:07:07.264228+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"111'531860","reported_seq":1062919,"reported_epoch":111,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:07:13.559351+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:07:13.559351+0000","last_peered":"2023-10-17T02:07:13.559351+0000","last_clean":"2023-10-17T02:07:13.559351+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:07:13.559351+0000","last_undegraded":"2023-10-17T02:07:13.559351+0000","last_fullsized":"2023-10-17T02:07:13.559351+0000","mapping_epoch":16,"log_start":"110'521818","ondisk_log_start":"110'521818","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":942,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 348s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7056289811,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":531745,"num_read_kb":2095240,"num_write":530065,"num_write_kb":2120254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"111'10626","reported_seq":14887,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:07:12.383286+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:07:12.383286+0000","last_peered":"2023-10-17T02:07:12.383286+0000","last_clean":"2023-10-17T02:07:12.383286+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:07:12.383286+0000","last_undegraded":"2023-10-17T02:07:12.383286+0000","last_fullsized":"2023-10-17T02:07:12.383286+0000","mapping_epoch":13,"log_start":"46'600","ondisk_log_start":"46'600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":290,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:06:32.401237+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:06:32.401237+0000","last_peered":"2023-10-17T02:06:32.401237+0000","last_clean":"2023-10-17T02:06:32.401237+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:06:32.401237+0000","last_undegraded":"2023-10-17T02:06:32.401237+0000","last_fullsized":"2023-10-17T02:06:32.401237+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28382597139,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2136651,"num_read_kb":8420796,"num_write":2133223,"num_write_kb":8532886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17470947328,"data_stored":20871807014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40207,"ondisk_log_size":40207,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":47456,"ondisk_log_size":47456,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739392,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9764124,"kb_used_data":8529172,"kb_used_omap":832,"kb_used_meta":1234111,"kb_avail":84607716,"statfs":{"total":96636764160,"available":86638301184,"internally_reserved":0,"allocated":8733872128,"data_stored":10433669908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":852977,"internal_metadata":1263729679},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:06:27 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.57699999999999996,"15min":0.46400000000000002},"min":{"1min":0.189,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":2.8479999999999999,"5min":12.852,"15min":12.852},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.439,"5min":0.64100000000000001,"15min":0.54400000000000004},"min":{"1min":0.20499999999999999,"5min":0.192,"15min":0.188},"max":{"1min":2.8090000000000002,"5min":12.784000000000001,"15min":12.784000000000001},"last":0.32300000000000001}]},{"osd":1,"last update":"Tue Oct 17 02:07:02 2023","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.47999999999999998,"15min":0.47699999999999998},"min":{"1min":0.22800000000000001,"5min":0.21199999999999999,"15min":0.20699999999999999},"max":{"1min":0.42299999999999999,"5min":6.202,"15min":9.5429999999999993},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.70999999999999996,"15min":0.59699999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23100000000000001},"max":{"1min":0.44900000000000001,"5min":12.824999999999999,"15min":12.824999999999999},"last":0.36099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739389,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5306896,"kb_used_data":4201864,"kb_used_omap":364,"kb_used_meta":1104659,"kb_avail":89064944,"statfs":{"total":96636764160,"available":91202502656,"internally_reserved":0,"allocated":4302708736,"data_stored":5140280115,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":372742,"internal_metadata":1131171834},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,5,1,12,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:06:24 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.54700000000000004,"15min":0.49299999999999999},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.16300000000000001},"max":{"1min":1.5820000000000001,"5min":6.8049999999999997,"15min":6.8049999999999997},"last":3.9980000000000002},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.59499999999999997,"15min":0.54300000000000004},"min":{"1min":0.249,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":1.613,"5min":6.8479999999999999,"15min":6.8479999999999999},"last":4.0629999999999997}]},{"osd":2,"last update":"Tue Oct 17 02:06:24 2023","interfaces":[{"interface":"back","average":{"1min":1.9239999999999999,"5min":0.81399999999999995,"15min":0.61399999999999999},"min":{"1min":0.23499999999999999,"5min":0.19500000000000001,"15min":0.188},"max":{"1min":25.128,"5min":25.128,"15min":25.128},"last":0.439},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.45000000000000001,"15min":0.47499999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.20100000000000001},"max":{"1min":1.4219999999999999,"5min":5.2089999999999996,"15min":6.7800000000000002},"last":0.46400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739389,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6106100,"kb_used_data":4333100,"kb_used_omap":416,"kb_used_meta":1772575,"kb_avail":88265740,"statfs":{"total":96636764160,"available":90384117760,"internally_reserved":0,"allocated":4437094400,"data_stored":5299511290,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":426086,"internal_metadata":1815117722},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:07:00 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.52600000000000002,"15min":0.45600000000000002},"min":{"1min":0.20000000000000001,"5min":0.182,"15min":0.17000000000000001},"max":{"1min":5.1130000000000004,"5min":6.1360000000000001,"15min":6.1360000000000001},"last":0.19600000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.625,"15min":0.503},"min":{"1min":0.218,"5min":0.191,"15min":0.191},"max":{"1min":1.7729999999999999,"5min":7.4880000000000004,"15min":7.4880000000000004},"last":0.25700000000000001}]},{"osd":2,"last update":"Tue Oct 17 02:07:06 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.68400000000000005,"15min":0.52300000000000002},"min":{"1min":0.23200000000000001,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":0.59699999999999998,"5min":10.125,"15min":10.125},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.61299999999999999,"15min":0.63200000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.20499999999999999},"max":{"1min":1.8009999999999999,"5min":6.3899999999999997,"15min":10.378},"last":0.40899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4435906560,"data_stored":5298503680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4301774848,"data_stored":5139607571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8733265920,"data_stored":10433695763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:07:13.631 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:07:13.631 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-17T02:07:14.239 INFO:teuthology.orchestra.run.smithi169.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-17T02:07:14.273 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:07:23.628 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:07:23.620+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T02:07:23.628 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:07:23.620+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T02:07:23.629 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:07:23.620+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbda750 2023-10-17T02:07:23.629 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:07:23.620+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T02:07:23.629 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:07:23.620+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T02:07:23.630 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:07:23.620+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T02:07:23.630 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:07:23.620+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T02:07:23.630 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:07:23.620+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T02:07:23.631 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:07:23.620+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T02:07:23.631 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:07:23.620+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T02:07:23.631 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:07:23.620+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T02:07:23.632 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:07:23.620+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T02:07:23.632 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:07:23.620+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T02:07:23.632 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:07:23.620+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T02:07:23.633 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:07:23.620+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T02:07:34.277 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:07:34.913 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:07:34.914 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:07:34.947 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2672,"stamp":"2023-10-17T02:07:33.571759+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28409498456,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2167549,"num_read_kb":8469089,"num_write":2140308,"num_write_kb":8560883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87870,"ondisk_log_size":87870,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21414004,"kb_used_data":17118872,"kb_used_omap":1613,"kb_used_meta":4293490,"kb_avail":261701516,"statfs":{"total":289910292480,"available":267982352384,"internally_reserved":0,"allocated":17529724928,"data_stored":20929510977,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1651805,"internal_metadata":4396534691},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,3,3,16,14,5,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18833408,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4770,"num_read_kb":19040,"num_write":4618,"num_write_kb":18472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002695"},"pg_stats":[{"pgid":"2.7","version":"111'14134","reported_seq":19897,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:07:27.747986+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:07:27.747986+0000","last_peered":"2023-10-17T02:07:27.747986+0000","last_clean":"2023-10-17T02:07:27.747986+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:07:27.747986+0000","last_undegraded":"2023-10-17T02:07:27.747986+0000","last_fullsized":"2023-10-17T02:07:27.747986+0000","mapping_epoch":13,"log_start":"79'4100","ondisk_log_start":"79'4100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"111'10646","reported_seq":16143,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:07:27.747443+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:07:27.747443+0000","last_peered":"2023-10-17T02:07:27.747443+0000","last_clean":"2023-10-17T02:07:27.747443+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:07:27.747443+0000","last_undegraded":"2023-10-17T02:07:27.747443+0000","last_fullsized":"2023-10-17T02:07:27.747443+0000","mapping_epoch":13,"log_start":"46'600","ondisk_log_start":"46'600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"111'7137","reported_seq":10110,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:07:27.747171+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:07:27.747171+0000","last_peered":"2023-10-17T02:07:27.747171+0000","last_clean":"2023-10-17T02:07:27.747171+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:07:27.747171+0000","last_undegraded":"2023-10-17T02:07:27.747171+0000","last_fullsized":"2023-10-17T02:07:27.747171+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":7137,"log_dups_size":0,"ondisk_log_size":7137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"111'10681","reported_seq":15587,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:07:27.747374+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:07:27.747374+0000","last_peered":"2023-10-17T02:07:27.747374+0000","last_clean":"2023-10-17T02:07:27.747374+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:07:27.747374+0000","last_undegraded":"2023-10-17T02:07:27.747374+0000","last_fullsized":"2023-10-17T02:07:27.747374+0000","mapping_epoch":13,"log_start":"46'600","ondisk_log_start":"46'600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"111'545582","reported_seq":1087787,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:07:33.020986+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:07:33.020986+0000","last_peered":"2023-10-17T02:07:33.020986+0000","last_clean":"2023-10-17T02:07:33.020986+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:07:33.020986+0000","last_undegraded":"2023-10-17T02:07:33.020986+0000","last_fullsized":"2023-10-17T02:07:33.020986+0000","mapping_epoch":16,"log_start":"110'535510","ondisk_log_start":"110'535510","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7203258368,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":542891,"num_read_kb":2138840,"num_write":542825,"num_write_kb":2171300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":269,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-17T02:06:02.679940+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:06:02.679940+0000","last_peered":"2023-10-17T02:06:02.679940+0000","last_clean":"2023-10-17T02:06:02.679940+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:06:02.679940+0000","last_undegraded":"2023-10-17T02:06:02.679940+0000","last_fullsized":"2023-10-17T02:06:02.679940+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"111'524168","reported_seq":1046245,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:07:28.616300+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:07:28.616300+0000","last_peered":"2023-10-17T02:07:28.616300+0000","last_clean":"2023-10-17T02:07:28.616300+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:07:28.616300+0000","last_undegraded":"2023-10-17T02:07:28.616300+0000","last_fullsized":"2023-10-17T02:07:28.616300+0000","mapping_epoch":16,"log_start":"110'514119","ondisk_log_start":"110'514119","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6923542528,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":522757,"num_read_kb":2061500,"num_write":521518,"num_write_kb":2086072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2936,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:07:27.748057+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:07:27.748057+0000","last_peered":"2023-10-17T02:07:27.748057+0000","last_clean":"2023-10-17T02:07:27.748057+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:07:27.748057+0000","last_undegraded":"2023-10-17T02:07:27.748057+0000","last_fullsized":"2023-10-17T02:07:27.748057+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"111'546628","reported_seq":1090575,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:07:33.010235+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:07:33.010235+0000","last_peered":"2023-10-17T02:07:33.010235+0000","last_clean":"2023-10-17T02:07:33.010235+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:07:33.010235+0000","last_undegraded":"2023-10-17T02:07:33.010235+0000","last_fullsized":"2023-10-17T02:07:33.010235+0000","mapping_epoch":16,"log_start":"110'536625","ondisk_log_start":"110'536625","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7220105216,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":544664,"num_read_kb":2146748,"num_write":543874,"num_write_kb":2175496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4401,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:07:27.747475+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:07:27.747475+0000","last_peered":"2023-10-17T02:07:27.747475+0000","last_clean":"2023-10-17T02:07:27.747475+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:07:27.747475+0000","last_undegraded":"2023-10-17T02:07:27.747475+0000","last_fullsized":"2023-10-17T02:07:27.747475+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"111'533289","reported_seq":1065889,"reported_epoch":111,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:07:28.639593+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:07:28.639593+0000","last_peered":"2023-10-17T02:07:28.639593+0000","last_clean":"2023-10-17T02:07:28.639593+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:07:28.639593+0000","last_undegraded":"2023-10-17T02:07:28.639593+0000","last_fullsized":"2023-10-17T02:07:28.639593+0000","mapping_epoch":16,"log_start":"110'523218","ondisk_log_start":"110'523218","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1000,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 363s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7061999635,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":533275,"num_read_kb":2101348,"num_write":531461,"num_write_kb":2125838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"111'10650","reported_seq":14920,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:07:27.747292+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:07:27.747292+0000","last_peered":"2023-10-17T02:07:27.747292+0000","last_clean":"2023-10-17T02:07:27.747292+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:07:27.747292+0000","last_undegraded":"2023-10-17T02:07:27.747292+0000","last_fullsized":"2023-10-17T02:07:27.747292+0000","mapping_epoch":13,"log_start":"46'600","ondisk_log_start":"46'600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":290,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:06:32.401237+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:06:32.401237+0000","last_peered":"2023-10-17T02:06:32.401237+0000","last_clean":"2023-10-17T02:06:32.401237+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:06:32.401237+0000","last_undegraded":"2023-10-17T02:06:32.401237+0000","last_fullsized":"2023-10-17T02:06:32.401237+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28408905747,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2143587,"num_read_kb":8448436,"num_write":2139678,"num_write_kb":8558706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17527959552,"data_stored":20928819238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40195,"ondisk_log_size":40195,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":47576,"ondisk_log_size":47576,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739396,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9874248,"kb_used_data":8558784,"kb_used_omap":832,"kb_used_meta":1314623,"kb_avail":84497592,"statfs":{"total":96636764160,"available":86525534208,"internally_reserved":0,"allocated":8764194816,"data_stored":10463992596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":852977,"internal_metadata":1346173967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,7,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:07:28 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.504,"15min":0.443},"min":{"1min":0.192,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":1.9810000000000001,"5min":12.852,"15min":12.852},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.56100000000000005,"15min":0.53400000000000003},"min":{"1min":0.221,"5min":0.192,"15min":0.188},"max":{"1min":0.50700000000000001,"5min":12.784000000000001,"15min":12.784000000000001},"last":0.45000000000000001}]},{"osd":1,"last update":"Tue Oct 17 02:07:02 2023","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.47999999999999998,"15min":0.47699999999999998},"min":{"1min":0.22800000000000001,"5min":0.21199999999999999,"15min":0.20699999999999999},"max":{"1min":0.42299999999999999,"5min":6.202,"15min":9.5429999999999993},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.70999999999999996,"15min":0.59699999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23100000000000001},"max":{"1min":0.44900000000000001,"5min":12.824999999999999,"15min":12.824999999999999},"last":0.54400000000000004}]}]},{"osd":1,"up_from":8,"seq":34359739392,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5362164,"kb_used_data":4212716,"kb_used_omap":364,"kb_used_meta":1149075,"kb_avail":89009676,"statfs":{"total":96636764160,"available":91145908224,"internally_reserved":0,"allocated":4313821184,"data_stored":5151392563,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":372742,"internal_metadata":1176653818},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,8,2,5,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:07:24 2023","interfaces":[{"interface":"back","average":{"1min":1.591,"5min":0.80800000000000005,"15min":0.55400000000000005},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16300000000000001},"max":{"1min":11.891999999999999,"5min":11.891999999999999,"15min":11.891999999999999},"last":3.0009999999999999},{"interface":"front","average":{"1min":1.631,"5min":0.85599999999999998,"15min":0.60299999999999998},"min":{"1min":0.252,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":11.926,"5min":11.926,"15min":11.926},"last":3.0409999999999999}]},{"osd":2,"last update":"Tue Oct 17 02:07:24 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.83799999999999997,"15min":0.62},"min":{"1min":0.24299999999999999,"5min":0.19500000000000001,"15min":0.188},"max":{"1min":8.0570000000000004,"5min":25.128,"15min":25.128},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.48899999999999999,"15min":0.46400000000000002},"min":{"1min":0.23200000000000001,"5min":0.22500000000000001,"15min":0.20100000000000001},"max":{"1min":3.9649999999999999,"5min":5.2089999999999996,"15min":6.7800000000000002},"last":0.40899999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739393,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6177592,"kb_used_data":4347372,"kb_used_omap":416,"kb_used_meta":1829791,"kb_avail":88194248,"statfs":{"total":96636764160,"available":90310909952,"internally_reserved":0,"allocated":4451708928,"data_stored":5314125818,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":426086,"internal_metadata":1873706906},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,0,1,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:07:00 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.52600000000000002,"15min":0.45600000000000002},"min":{"1min":0.20000000000000001,"5min":0.182,"15min":0.17000000000000001},"max":{"1min":5.1130000000000004,"5min":6.1360000000000001,"15min":6.1360000000000001},"last":0.20200000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.625,"15min":0.503},"min":{"1min":0.218,"5min":0.191,"15min":0.191},"max":{"1min":1.7729999999999999,"5min":7.4880000000000004,"15min":7.4880000000000004},"last":5.5119999999999996}]},{"osd":2,"last update":"Tue Oct 17 02:07:06 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.68400000000000005,"15min":0.52300000000000002},"min":{"1min":0.23200000000000001,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":0.59699999999999998,"5min":10.125,"15min":10.125},"last":5.6580000000000004},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.61299999999999999,"15min":0.63200000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.20499999999999999},"max":{"1min":1.8009999999999999,"5min":6.3899999999999997,"15min":10.378},"last":5.5529999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4451135488,"data_stored":5313732608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4313010176,"data_stored":5150842899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8763813888,"data_stored":10464243731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:07:34.949 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:07:34.950 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:07:54.951 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:07:55.806 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:07:55.807 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:07:55.852 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2684,"stamp":"2023-10-17T02:07:55.577225+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28443913048,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2176209,"num_read_kb":8503665,"num_write":2148746,"num_write_kb":8594635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87817,"ondisk_log_size":87817,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21677864,"kb_used_data":17182856,"kb_used_omap":1613,"kb_used_meta":4493362,"kb_avail":261437656,"statfs":{"total":289910292480,"available":267712159744,"internally_reserved":0,"allocated":17595244544,"data_stored":20995017003,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1651805,"internal_metadata":4601203619},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,9,26,27],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19222528,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4773,"num_read_kb":19048,"num_write":4708,"num_write_kb":18832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002079"},"pg_stats":[{"pgid":"2.7","version":"112'14206","reported_seq":19990,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:07:53.347805+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:07:53.347805+0000","last_peered":"2023-10-17T02:07:53.347805+0000","last_clean":"2023-10-17T02:07:53.347805+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:07:53.347805+0000","last_undegraded":"2023-10-17T02:07:53.347805+0000","last_fullsized":"2023-10-17T02:07:53.347805+0000","mapping_epoch":13,"log_start":"81'4200","ondisk_log_start":"81'4200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"112'10700","reported_seq":16213,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:07:53.347563+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:07:53.347563+0000","last_peered":"2023-10-17T02:07:53.347563+0000","last_clean":"2023-10-17T02:07:53.347563+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:07:53.347563+0000","last_undegraded":"2023-10-17T02:07:53.347563+0000","last_fullsized":"2023-10-17T02:07:53.347563+0000","mapping_epoch":13,"log_start":"46'600","ondisk_log_start":"46'600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"112'7173","reported_seq":10157,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:07:53.347269+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:07:53.347269+0000","last_peered":"2023-10-17T02:07:53.347269+0000","last_clean":"2023-10-17T02:07:53.347269+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:07:53.347269+0000","last_undegraded":"2023-10-17T02:07:53.347269+0000","last_fullsized":"2023-10-17T02:07:53.347269+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":7173,"log_dups_size":0,"ondisk_log_size":7173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"112'10735","reported_seq":15657,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:07:53.347401+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:07:53.347401+0000","last_peered":"2023-10-17T02:07:53.347401+0000","last_clean":"2023-10-17T02:07:53.347401+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:07:53.347401+0000","last_undegraded":"2023-10-17T02:07:53.347401+0000","last_fullsized":"2023-10-17T02:07:53.347401+0000","mapping_epoch":13,"log_start":"46'700","ondisk_log_start":"46'700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"112'547538","reported_seq":1091696,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:07:53.087623+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:07:53.087623+0000","last_peered":"2023-10-17T02:07:53.087623+0000","last_clean":"2023-10-17T02:07:53.087623+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:07:53.087623+0000","last_undegraded":"2023-10-17T02:07:53.087623+0000","last_fullsized":"2023-10-17T02:07:53.087623+0000","mapping_epoch":16,"log_start":"110'537510","ondisk_log_start":"110'537510","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7211241472,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":544840,"num_read_kb":2146632,"num_write":544781,"num_write_kb":2179124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":270,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:06:32.415535+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:06:32.415535+0000","last_peered":"2023-10-17T02:06:32.415535+0000","last_clean":"2023-10-17T02:06:32.415535+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:06:32.415535+0000","last_undegraded":"2023-10-17T02:06:32.415535+0000","last_fullsized":"2023-10-17T02:06:32.415535+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"112'526489","reported_seq":1050934,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:07:53.788988+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:07:53.788988+0000","last_peered":"2023-10-17T02:07:53.788988+0000","last_clean":"2023-10-17T02:07:53.788988+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:07:53.788988+0000","last_undegraded":"2023-10-17T02:07:53.788988+0000","last_fullsized":"2023-10-17T02:07:53.788988+0000","mapping_epoch":16,"log_start":"110'516419","ondisk_log_start":"110'516419","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6933000192,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":525124,"num_read_kb":2070952,"num_write":523839,"num_write_kb":2095356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2945,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:07:48.360920+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:07:48.360920+0000","last_peered":"2023-10-17T02:07:48.360920+0000","last_clean":"2023-10-17T02:07:48.360920+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:07:48.360920+0000","last_undegraded":"2023-10-17T02:07:48.360920+0000","last_fullsized":"2023-10-17T02:07:48.360920+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"112'548480","reported_seq":1094429,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:07:53.087622+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:07:53.087622+0000","last_peered":"2023-10-17T02:07:53.087622+0000","last_clean":"2023-10-17T02:07:53.087622+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:07:53.087622+0000","last_undegraded":"2023-10-17T02:07:53.087622+0000","last_fullsized":"2023-10-17T02:07:53.087622+0000","mapping_epoch":16,"log_start":"110'538425","ondisk_log_start":"110'538425","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7227666432,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":546664,"num_read_kb":2154732,"num_write":545726,"num_write_kb":2182904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4414,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:07:48.223382+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:07:48.223382+0000","last_peered":"2023-10-17T02:07:48.223382+0000","last_clean":"2023-10-17T02:07:48.223382+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:07:48.223382+0000","last_undegraded":"2023-10-17T02:07:48.223382+0000","last_fullsized":"2023-10-17T02:07:48.223382+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"112'535637","reported_seq":1070590,"reported_epoch":112,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:07:53.793789+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:07:53.793789+0000","last_peered":"2023-10-17T02:07:53.793789+0000","last_clean":"2023-10-17T02:07:53.793789+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:07:53.793789+0000","last_undegraded":"2023-10-17T02:07:53.793789+0000","last_fullsized":"2023-10-17T02:07:53.793789+0000","mapping_epoch":16,"log_start":"110'525618","ondisk_log_start":"110'525618","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1072,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 388s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7071412243,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":535619,"num_read_kb":2110696,"num_write":533770,"num_write_kb":2135074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"112'10704","reported_seq":14990,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:07:53.347416+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:07:53.347416+0000","last_peered":"2023-10-17T02:07:53.347416+0000","last_clean":"2023-10-17T02:07:53.347416+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:07:53.347416+0000","last_undegraded":"2023-10-17T02:07:53.347416+0000","last_fullsized":"2023-10-17T02:07:53.347416+0000","mapping_epoch":13,"log_start":"48'700","ondisk_log_start":"48'700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":291,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:06:43.373953+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:06:43.373953+0000","last_peered":"2023-10-17T02:06:43.373953+0000","last_clean":"2023-10-17T02:06:43.373953+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:06:43.373953+0000","last_undegraded":"2023-10-17T02:06:43.373953+0000","last_fullsized":"2023-10-17T02:06:43.373953+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28443320339,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2152247,"num_read_kb":8483012,"num_write":2148116,"num_write_kb":8592458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17593057280,"data_stored":20993916966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40172,"ondisk_log_size":40172,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":47546,"ondisk_log_size":47546,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739400,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9989976,"kb_used_data":8588428,"kb_used_omap":832,"kb_used_meta":1400703,"kb_avail":84381864,"statfs":{"total":96636764160,"available":86407028736,"internally_reserved":0,"allocated":8794550272,"data_stored":10494343522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":852977,"internal_metadata":1434319887},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,4,10,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:07:28 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.504,"15min":0.443},"min":{"1min":0.192,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":1.9810000000000001,"5min":12.852,"15min":12.852},"last":0.219},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.56100000000000005,"15min":0.53400000000000003},"min":{"1min":0.221,"5min":0.192,"15min":0.188},"max":{"1min":0.50700000000000001,"5min":12.784000000000001,"15min":12.784000000000001},"last":0.24099999999999999}]},{"osd":1,"last update":"Tue Oct 17 02:07:02 2023","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.47999999999999998,"15min":0.47699999999999998},"min":{"1min":0.22800000000000001,"5min":0.21199999999999999,"15min":0.20699999999999999},"max":{"1min":0.42299999999999999,"5min":6.202,"15min":9.5429999999999993},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.70999999999999996,"15min":0.59699999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23100000000000001},"max":{"1min":0.44900000000000001,"5min":12.824999999999999,"15min":12.824999999999999},"last":0.45500000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739397,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5458372,"kb_used_data":4231224,"kb_used_omap":364,"kb_used_meta":1226771,"kb_avail":88913468,"statfs":{"total":96636764160,"available":91047391232,"internally_reserved":0,"allocated":4332773376,"data_stored":5170340225,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":372742,"internal_metadata":1256214522},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,3,11,19],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:07:24 2023","interfaces":[{"interface":"back","average":{"1min":1.591,"5min":0.80800000000000005,"15min":0.55400000000000005},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16300000000000001},"max":{"1min":11.891999999999999,"5min":11.891999999999999,"15min":11.891999999999999},"last":0.26900000000000002},{"interface":"front","average":{"1min":1.631,"5min":0.85599999999999998,"15min":0.60299999999999998},"min":{"1min":0.252,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":11.926,"5min":11.926,"15min":11.926},"last":0.29799999999999999}]},{"osd":2,"last update":"Tue Oct 17 02:07:24 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.83799999999999997,"15min":0.62},"min":{"1min":0.24299999999999999,"5min":0.19500000000000001,"15min":0.188},"max":{"1min":8.0570000000000004,"5min":25.128,"15min":25.128},"last":2.2389999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.48899999999999999,"15min":0.46400000000000002},"min":{"1min":0.23200000000000001,"5min":0.22500000000000001,"15min":0.20100000000000001},"max":{"1min":3.9649999999999999,"5min":5.2089999999999996,"15min":6.7800000000000002},"last":0.23699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739397,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6229516,"kb_used_data":4363204,"kb_used_omap":416,"kb_used_meta":1865887,"kb_avail":88142324,"statfs":{"total":96636764160,"available":90257739776,"internally_reserved":0,"allocated":4467920896,"data_stored":5330333256,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":426086,"internal_metadata":1910669210},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:07:00 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.52600000000000002,"15min":0.45600000000000002},"min":{"1min":0.20000000000000001,"5min":0.182,"15min":0.17000000000000001},"max":{"1min":5.1130000000000004,"5min":6.1360000000000001,"15min":6.1360000000000001},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.625,"15min":0.503},"min":{"1min":0.218,"5min":0.191,"15min":0.191},"max":{"1min":1.7729999999999999,"5min":7.4880000000000004,"15min":7.4880000000000004},"last":0.27900000000000003}]},{"osd":2,"last update":"Tue Oct 17 02:07:06 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.68400000000000005,"15min":0.52300000000000002},"min":{"1min":0.23200000000000001,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":0.59699999999999998,"5min":10.125,"15min":10.125},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.61299999999999999,"15min":0.63200000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.20499999999999999},"max":{"1min":1.8009999999999999,"5min":6.3899999999999997,"15min":10.378},"last":0.36199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4466737152,"data_stored":5329334272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4331970560,"data_stored":5169803283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8794349568,"data_stored":10494779411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:07:55.854 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:07:55.855 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:08:15.855 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:08:16.354 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:08:16.355 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:08:16.383 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2694,"stamp":"2023-10-17T02:08:15.581350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28474272600,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2183843,"num_read_kb":8534085,"num_write":2156183,"num_write_kb":8624383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87890,"ondisk_log_size":87890,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21918612,"kb_used_data":17241340,"kb_used_omap":1613,"kb_used_meta":4675634,"kb_avail":261196908,"statfs":{"total":289910292480,"available":267465633792,"internally_reserved":0,"allocated":17655132160,"data_stored":21054904619,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1651805,"internal_metadata":4787850147},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,7,11,5,14,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18976768,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4759,"num_read_kb":18960,"num_write":4650,"num_write_kb":18600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002366"},"pg_stats":[{"pgid":"2.7","version":"112'14258","reported_seq":20058,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:08:13.823936+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:08:13.823936+0000","last_peered":"2023-10-17T02:08:13.823936+0000","last_clean":"2023-10-17T02:08:13.823936+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:08:13.823936+0000","last_undegraded":"2023-10-17T02:08:13.823936+0000","last_fullsized":"2023-10-17T02:08:13.823936+0000","mapping_epoch":13,"log_start":"81'4200","ondisk_log_start":"81'4200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"112'10739","reported_seq":16264,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:08:13.823728+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:08:13.823728+0000","last_peered":"2023-10-17T02:08:13.823728+0000","last_clean":"2023-10-17T02:08:13.823728+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:08:13.823728+0000","last_undegraded":"2023-10-17T02:08:13.823728+0000","last_fullsized":"2023-10-17T02:08:13.823728+0000","mapping_epoch":13,"log_start":"46'700","ondisk_log_start":"46'700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"112'7199","reported_seq":10192,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:08:13.823127+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:08:13.823127+0000","last_peered":"2023-10-17T02:08:13.823127+0000","last_clean":"2023-10-17T02:08:13.823127+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:08:13.823127+0000","last_undegraded":"2023-10-17T02:08:13.823127+0000","last_fullsized":"2023-10-17T02:08:13.823127+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":7199,"log_dups_size":0,"ondisk_log_size":7199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"112'10774","reported_seq":15709,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:08:13.823263+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:08:13.823263+0000","last_peered":"2023-10-17T02:08:13.823263+0000","last_clean":"2023-10-17T02:08:13.823263+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:08:13.823263+0000","last_undegraded":"2023-10-17T02:08:13.823263+0000","last_fullsized":"2023-10-17T02:08:13.823263+0000","mapping_epoch":13,"log_start":"46'700","ondisk_log_start":"46'700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"112'549357","reported_seq":1095453,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:08:13.237316+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:08:13.237316+0000","last_peered":"2023-10-17T02:08:13.237316+0000","last_clean":"2023-10-17T02:08:13.237316+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:08:13.237316+0000","last_undegraded":"2023-10-17T02:08:13.237316+0000","last_fullsized":"2023-10-17T02:08:13.237316+0000","mapping_epoch":16,"log_start":"110'539310","ondisk_log_start":"110'539310","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7218634752,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":546783,"num_read_kb":2154372,"num_write":546600,"num_write_kb":2186400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":271,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:07:37.436799+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:07:37.436799+0000","last_peered":"2023-10-17T02:07:37.436799+0000","last_clean":"2023-10-17T02:07:37.436799+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:07:37.436799+0000","last_undegraded":"2023-10-17T02:07:37.436799+0000","last_fullsized":"2023-10-17T02:07:37.436799+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"112'528320","reported_seq":1054607,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:08:13.908537+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:08:13.908537+0000","last_peered":"2023-10-17T02:08:13.908537+0000","last_clean":"2023-10-17T02:08:13.908537+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:08:13.908537+0000","last_undegraded":"2023-10-17T02:08:13.908537+0000","last_fullsized":"2023-10-17T02:08:13.908537+0000","mapping_epoch":16,"log_start":"110'518319","ondisk_log_start":"110'518319","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6940459008,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":526967,"num_read_kb":2078288,"num_write":525670,"num_write_kb":2102680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2954,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:08:08.703257+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:08:08.703257+0000","last_peered":"2023-10-17T02:08:08.703257+0000","last_clean":"2023-10-17T02:08:08.703257+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:08:08.703257+0000","last_undegraded":"2023-10-17T02:08:08.703257+0000","last_fullsized":"2023-10-17T02:08:08.703257+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"112'550397","reported_seq":1098244,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:08:13.256795+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:08:13.256795+0000","last_peered":"2023-10-17T02:08:13.256795+0000","last_clean":"2023-10-17T02:08:13.256795+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:08:13.256795+0000","last_undegraded":"2023-10-17T02:08:13.256795+0000","last_fullsized":"2023-10-17T02:08:13.256795+0000","mapping_epoch":16,"log_start":"110'540325","ondisk_log_start":"110'540325","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7235502080,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":548562,"num_read_kb":2162304,"num_write":547643,"num_write_kb":2190572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4426,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:08:08.706985+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:08:08.706985+0000","last_peered":"2023-10-17T02:08:08.706985+0000","last_clean":"2023-10-17T02:08:08.706985+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:08:08.706985+0000","last_undegraded":"2023-10-17T02:08:08.706985+0000","last_fullsized":"2023-10-17T02:08:08.706985+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"112'537548","reported_seq":1074461,"reported_epoch":112,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:08:13.916543+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:08:13.916543+0000","last_peered":"2023-10-17T02:08:13.916543+0000","last_clean":"2023-10-17T02:08:13.916543+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:08:13.916543+0000","last_undegraded":"2023-10-17T02:08:13.916543+0000","last_fullsized":"2023-10-17T02:08:13.916543+0000","mapping_epoch":16,"log_start":"110'527518","ondisk_log_start":"110'527518","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1140,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 408s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7079084051,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":537569,"num_read_kb":2118468,"num_write":535640,"num_write_kb":2142554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"112'10743","reported_seq":15042,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:08:13.823368+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:08:13.823368+0000","last_peered":"2023-10-17T02:08:13.823368+0000","last_clean":"2023-10-17T02:08:13.823368+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:08:13.823368+0000","last_undegraded":"2023-10-17T02:08:13.823368+0000","last_fullsized":"2023-10-17T02:08:13.823368+0000","mapping_epoch":13,"log_start":"48'700","ondisk_log_start":"48'700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":291,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:06:43.373953+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:06:43.373953+0000","last_peered":"2023-10-17T02:06:43.373953+0000","last_clean":"2023-10-17T02:06:43.373953+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:06:43.373953+0000","last_undegraded":"2023-10-17T02:06:43.373953+0000","last_fullsized":"2023-10-17T02:06:43.373953+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28473679891,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2159881,"num_read_kb":8513432,"num_write":2155553,"num_write_kb":8622206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17653936128,"data_stored":21054795814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40150,"ondisk_log_size":40150,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":47641,"ondisk_log_size":47641,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739404,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10113064,"kb_used_data":8618080,"kb_used_omap":832,"kb_used_meta":1494143,"kb_avail":84258776,"statfs":{"total":96636764160,"available":86280986624,"internally_reserved":0,"allocated":8824913920,"data_stored":10524707170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":852977,"internal_metadata":1530002447},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:07:28 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.504,"15min":0.443},"min":{"1min":0.192,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":1.9810000000000001,"5min":12.852,"15min":12.852},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.56100000000000005,"15min":0.53400000000000003},"min":{"1min":0.221,"5min":0.192,"15min":0.188},"max":{"1min":0.50700000000000001,"5min":12.784000000000001,"15min":12.784000000000001},"last":0.39400000000000002}]},{"osd":1,"last update":"Tue Oct 17 02:08:03 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52500000000000002,"15min":0.49299999999999999},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":4.6980000000000004,"5min":6.202,"15min":9.5429999999999993},"last":0.249},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.64400000000000002,"15min":0.60499999999999998},"min":{"1min":0.27200000000000002,"5min":0.23799999999999999,"15min":0.23100000000000001},"max":{"1min":4.6550000000000002,"5min":12.824999999999999,"15min":12.824999999999999},"last":0.28199999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739401,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5525656,"kb_used_data":4245776,"kb_used_omap":364,"kb_used_meta":1279507,"kb_avail":88846184,"statfs":{"total":96636764160,"available":90978492416,"internally_reserved":0,"allocated":4347674624,"data_stored":5185241473,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":372742,"internal_metadata":1310216186},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,3,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:07:24 2023","interfaces":[{"interface":"back","average":{"1min":1.591,"5min":0.80800000000000005,"15min":0.55400000000000005},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16300000000000001},"max":{"1min":11.891999999999999,"5min":11.891999999999999,"15min":11.891999999999999},"last":0.23300000000000001},{"interface":"front","average":{"1min":1.631,"5min":0.85599999999999998,"15min":0.60299999999999998},"min":{"1min":0.252,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":11.926,"5min":11.926,"15min":11.926},"last":0.28000000000000003}]},{"osd":2,"last update":"Tue Oct 17 02:07:24 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.83799999999999997,"15min":0.62},"min":{"1min":0.24299999999999999,"5min":0.19500000000000001,"15min":0.188},"max":{"1min":8.0570000000000004,"5min":25.128,"15min":25.128},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.48899999999999999,"15min":0.46400000000000002},"min":{"1min":0.23200000000000001,"5min":0.22500000000000001,"15min":0.20100000000000001},"max":{"1min":3.9649999999999999,"5min":5.2089999999999996,"15min":6.7800000000000002},"last":0.214}]}]},{"osd":0,"up_from":8,"seq":34359739401,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6279892,"kb_used_data":4377484,"kb_used_omap":416,"kb_used_meta":1901983,"kb_avail":88091948,"statfs":{"total":96636764160,"available":90206154752,"internally_reserved":0,"allocated":4482543616,"data_stored":5344955976,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":426086,"internal_metadata":1947631514},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,3,1,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:08:02 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.55800000000000005,"15min":0.46600000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":4.0990000000000002,"5min":6.1360000000000001,"15min":6.1360000000000001},"last":0.439},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.67100000000000004,"15min":0.497},"min":{"1min":0.20599999999999999,"5min":0.191,"15min":0.191},"max":{"1min":5.5119999999999996,"5min":7.4880000000000004,"15min":7.4880000000000004},"last":0.48099999999999998}]},{"osd":2,"last update":"Tue Oct 17 02:08:09 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.70699999999999996,"15min":0.54100000000000004},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":5.6580000000000004,"5min":10.125,"15min":10.125},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.622,"15min":0.60799999999999998},"min":{"1min":0.245,"5min":0.23699999999999999,"15min":0.20499999999999999},"max":{"1min":5.5529999999999999,"5min":6.3899999999999997,"15min":10.378},"last":0.51300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4482035712,"data_stored":5344632832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4347129856,"data_stored":5184962579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8824770560,"data_stored":10525200403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:08:16.385 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:08:16.386 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:08:36.387 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:08:36.955 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:08:36.956 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:08:36.989 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2704,"stamp":"2023-10-17T02:08:35.588021+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28504349528,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2191679,"num_read_kb":8565361,"num_write":2163528,"num_write_kb":8653763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88101,"ondisk_log_size":88101,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22218104,"kb_used_data":17300572,"kb_used_omap":1613,"kb_used_meta":4915890,"kb_avail":260897416,"statfs":{"total":289910292480,"available":267158953984,"internally_reserved":0,"allocated":17715785728,"data_stored":21115558187,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1651805,"internal_metadata":5033872291},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,4,8,5,2,23],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18935808,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4930,"num_read_kb":19684,"num_write":4616,"num_write_kb":18464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005270"},"pg_stats":[{"pgid":"2.7","version":"112'14322","reported_seq":20134,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:08:33.931298+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:08:33.931298+0000","last_peered":"2023-10-17T02:08:33.931298+0000","last_clean":"2023-10-17T02:08:33.931298+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:08:33.931298+0000","last_undegraded":"2023-10-17T02:08:33.931298+0000","last_fullsized":"2023-10-17T02:08:33.931298+0000","mapping_epoch":13,"log_start":"81'4300","ondisk_log_start":"81'4300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"112'10787","reported_seq":16321,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:08:33.919256+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:08:33.919256+0000","last_peered":"2023-10-17T02:08:33.919256+0000","last_clean":"2023-10-17T02:08:33.919256+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:08:33.919256+0000","last_undegraded":"2023-10-17T02:08:33.919256+0000","last_fullsized":"2023-10-17T02:08:33.919256+0000","mapping_epoch":13,"log_start":"46'700","ondisk_log_start":"46'700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"112'7231","reported_seq":10230,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:08:33.910949+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:08:33.910949+0000","last_peered":"2023-10-17T02:08:33.910949+0000","last_clean":"2023-10-17T02:08:33.910949+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:08:33.910949+0000","last_undegraded":"2023-10-17T02:08:33.910949+0000","last_fullsized":"2023-10-17T02:08:33.910949+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":7231,"log_dups_size":0,"ondisk_log_size":7231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"112'10822","reported_seq":15766,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:08:33.926712+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:08:33.926712+0000","last_peered":"2023-10-17T02:08:33.926712+0000","last_clean":"2023-10-17T02:08:33.926712+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:08:33.926712+0000","last_undegraded":"2023-10-17T02:08:33.926712+0000","last_fullsized":"2023-10-17T02:08:33.926712+0000","mapping_epoch":13,"log_start":"48'800","ondisk_log_start":"48'800","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"112'551198","reported_seq":1099289,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:08:33.390896+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:08:33.390896+0000","last_peered":"2023-10-17T02:08:33.390896+0000","last_clean":"2023-10-17T02:08:33.390896+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:08:33.390896+0000","last_undegraded":"2023-10-17T02:08:33.390896+0000","last_fullsized":"2023-10-17T02:08:33.390896+0000","mapping_epoch":16,"log_start":"111'541110","ondisk_log_start":"111'541110","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7226179584,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":548775,"num_read_kb":2162324,"num_write":548441,"num_write_kb":2193764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":271,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:07:37.436799+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:07:37.436799+0000","last_peered":"2023-10-17T02:07:37.436799+0000","last_clean":"2023-10-17T02:07:37.436799+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:07:37.436799+0000","last_undegraded":"2023-10-17T02:07:37.436799+0000","last_fullsized":"2023-10-17T02:07:37.436799+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"112'530115","reported_seq":1058325,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:08:34.064658+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:08:34.064658+0000","last_peered":"2023-10-17T02:08:34.064658+0000","last_clean":"2023-10-17T02:08:34.064658+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:08:34.064658+0000","last_undegraded":"2023-10-17T02:08:34.064658+0000","last_fullsized":"2023-10-17T02:08:34.064658+0000","mapping_epoch":16,"log_start":"111'520019","ondisk_log_start":"111'520019","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6947844096,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":528892,"num_read_kb":2085976,"num_write":527465,"num_write_kb":2109860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2962,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:08:29.476115+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:08:29.476115+0000","last_peered":"2023-10-17T02:08:29.476115+0000","last_clean":"2023-10-17T02:08:29.476115+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:08:29.476115+0000","last_undegraded":"2023-10-17T02:08:29.476115+0000","last_fullsized":"2023-10-17T02:08:29.476115+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"112'552267","reported_seq":1102091,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:08:33.358865+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:08:33.358865+0000","last_peered":"2023-10-17T02:08:33.358865+0000","last_clean":"2023-10-17T02:08:33.358865+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:08:33.358865+0000","last_undegraded":"2023-10-17T02:08:33.358865+0000","last_fullsized":"2023-10-17T02:08:33.358865+0000","mapping_epoch":16,"log_start":"111'542225","ondisk_log_start":"111'542225","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7243153408,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":550541,"num_read_kb":2170192,"num_write":549513,"num_write_kb":2198052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4438,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:08:29.185273+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:08:29.185273+0000","last_peered":"2023-10-17T02:08:29.185273+0000","last_clean":"2023-10-17T02:08:29.185273+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:08:29.185273+0000","last_undegraded":"2023-10-17T02:08:29.185273+0000","last_fullsized":"2023-10-17T02:08:29.185273+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"112'539413","reported_seq":1078276,"reported_epoch":112,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:08:34.055490+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:08:34.055490+0000","last_peered":"2023-10-17T02:08:34.055490+0000","last_clean":"2023-10-17T02:08:34.055490+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:08:34.055490+0000","last_undegraded":"2023-10-17T02:08:34.055490+0000","last_fullsized":"2023-10-17T02:08:34.055490+0000","mapping_epoch":16,"log_start":"111'529318","ondisk_log_start":"111'529318","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1188,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 428s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7086579731,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":539509,"num_read_kb":2126216,"num_write":537479,"num_write_kb":2149910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"112'10791","reported_seq":15099,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:08:33.949211+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:08:33.949211+0000","last_peered":"2023-10-17T02:08:33.949211+0000","last_clean":"2023-10-17T02:08:33.949211+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:08:33.949211+0000","last_undegraded":"2023-10-17T02:08:33.949211+0000","last_fullsized":"2023-10-17T02:08:33.949211+0000","mapping_epoch":13,"log_start":"48'700","ondisk_log_start":"48'700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":291,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-17T02:06:43.373953+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:06:43.373953+0000","last_peered":"2023-10-17T02:06:43.373953+0000","last_clean":"2023-10-17T02:06:43.373953+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:06:43.373953+0000","last_undegraded":"2023-10-17T02:06:43.373953+0000","last_fullsized":"2023-10-17T02:06:43.373953+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28503756819,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2167717,"num_read_kb":8544708,"num_write":2162898,"num_write_kb":8651586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17714249728,"data_stored":21115109414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40321,"ondisk_log_size":40321,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":47681,"ondisk_log_size":47681,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739408,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10263696,"kb_used_data":8647364,"kb_used_omap":832,"kb_used_meta":1615487,"kb_avail":84108144,"statfs":{"total":96636764160,"available":86126739456,"internally_reserved":0,"allocated":8854900736,"data_stored":10554693986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":852977,"internal_metadata":1654258703},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:08:29 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.52000000000000002,"15min":0.435},"min":{"1min":0.184,"5min":0.184,"15min":0.16600000000000001},"max":{"1min":1.518,"5min":12.852,"15min":12.852},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.311,"5min":0.51800000000000002,"15min":0.51200000000000001},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.192},"max":{"1min":0.51700000000000002,"5min":12.784000000000001,"15min":12.784000000000001},"last":0.23799999999999999}]},{"osd":1,"last update":"Tue Oct 17 02:08:03 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52500000000000002,"15min":0.49299999999999999},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":4.6980000000000004,"5min":6.202,"15min":9.5429999999999993},"last":0.374},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.64400000000000002,"15min":0.60499999999999998},"min":{"1min":0.27200000000000002,"5min":0.23799999999999999,"15min":0.23100000000000001},"max":{"1min":4.6550000000000002,"5min":12.824999999999999,"15min":12.824999999999999},"last":0.308}]}]},{"osd":1,"up_from":8,"seq":34359739405,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5598056,"kb_used_data":4260896,"kb_used_omap":364,"kb_used_meta":1336787,"kb_avail":88773784,"statfs":{"total":96636764160,"available":90904354816,"internally_reserved":0,"allocated":4363157504,"data_stored":5200724353,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":372742,"internal_metadata":1368870906},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,1,2,23],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:08:27 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.76500000000000001,"15min":0.56599999999999995},"min":{"1min":0.17699999999999999,"5min":0.16900000000000001,"15min":0.16300000000000001},"max":{"1min":4.2480000000000002,"5min":11.891999999999999,"15min":11.891999999999999},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.80900000000000005,"15min":0.61399999999999999},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.22},"max":{"1min":4.2869999999999999,"5min":11.926,"15min":11.926},"last":0.34799999999999998}]},{"osd":2,"last update":"Tue Oct 17 02:08:27 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.84299999999999997,"15min":0.63400000000000001},"min":{"1min":0.20300000000000001,"5min":0.19500000000000001,"15min":0.192},"max":{"1min":2.8279999999999998,"5min":25.128,"15min":25.128},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.45900000000000002,"15min":0.46500000000000002},"min":{"1min":0.214,"5min":0.214,"15min":0.20100000000000001},"max":{"1min":2.669,"5min":5.2089999999999996,"15min":6.7800000000000002},"last":0.29299999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739405,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6356352,"kb_used_data":4392312,"kb_used_omap":416,"kb_used_meta":1963615,"kb_avail":88015488,"statfs":{"total":96636764160,"available":90127859712,"internally_reserved":0,"allocated":4497727488,"data_stored":5360139848,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":426086,"internal_metadata":2010742682},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:08:02 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.55800000000000005,"15min":0.46600000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":4.0990000000000002,"5min":6.1360000000000001,"15min":6.1360000000000001},"last":5.4370000000000003},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.67100000000000004,"15min":0.497},"min":{"1min":0.20599999999999999,"5min":0.191,"15min":0.191},"max":{"1min":5.5119999999999996,"5min":7.4880000000000004,"15min":7.4880000000000004},"last":0.26600000000000001}]},{"osd":2,"last update":"Tue Oct 17 02:08:09 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.70699999999999996,"15min":0.54100000000000004},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":5.6580000000000004,"5min":10.125,"15min":10.125},"last":0.248},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.622,"15min":0.60799999999999998},"min":{"1min":0.245,"5min":0.23699999999999999,"15min":0.20499999999999999},"max":{"1min":5.5529999999999999,"5min":6.3899999999999997,"15min":10.378},"last":0.29399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4497240064,"data_stored":5359837184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4362014720,"data_stored":5199847443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8854994944,"data_stored":10555424787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:08:36.994 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:08:36.994 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:08:56.995 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:08:57.488 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:08:57.490 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:08:57.525 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2715,"stamp":"2023-10-17T02:08:55.591449+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28535114584,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2199365,"num_read_kb":8596029,"num_write":2171098,"num_write_kb":8684043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87942,"ondisk_log_size":87942,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22003892,"kb_used_data":17361176,"kb_used_omap":1614,"kb_used_meta":4641073,"kb_avail":261111628,"statfs":{"total":289910292480,"available":267378307072,"internally_reserved":0,"allocated":17777844224,"data_stored":21177602964,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1653485,"internal_metadata":4752459027},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,4,4,7,15,14,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15503360,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3776,"num_read_kb":15056,"num_write":3814,"num_write_kb":15256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001747"},"pg_stats":[{"pgid":"2.7","version":"113'14358","reported_seq":20187,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:08:51.417319+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:08:51.417319+0000","last_peered":"2023-10-17T02:08:51.417319+0000","last_clean":"2023-10-17T02:08:51.417319+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:08:51.417319+0000","last_undegraded":"2023-10-17T02:08:51.417319+0000","last_fullsized":"2023-10-17T02:08:51.417319+0000","mapping_epoch":13,"log_start":"81'4300","ondisk_log_start":"81'4300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"113'10814","reported_seq":16361,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:08:51.417193+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:08:51.417193+0000","last_peered":"2023-10-17T02:08:51.417193+0000","last_clean":"2023-10-17T02:08:51.417193+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:08:51.417193+0000","last_undegraded":"2023-10-17T02:08:51.417193+0000","last_fullsized":"2023-10-17T02:08:51.417193+0000","mapping_epoch":13,"log_start":"48'800","ondisk_log_start":"48'800","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"113'7249","reported_seq":10257,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:08:51.392288+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:08:51.392288+0000","last_peered":"2023-10-17T02:08:51.392288+0000","last_clean":"2023-10-17T02:08:51.392288+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:08:51.392288+0000","last_undegraded":"2023-10-17T02:08:51.392288+0000","last_fullsized":"2023-10-17T02:08:51.392288+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":7249,"log_dups_size":0,"ondisk_log_size":7249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"113'10849","reported_seq":15806,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:08:51.417221+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:08:51.417221+0000","last_peered":"2023-10-17T02:08:51.417221+0000","last_clean":"2023-10-17T02:08:51.417221+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:08:51.417221+0000","last_undegraded":"2023-10-17T02:08:51.417221+0000","last_fullsized":"2023-10-17T02:08:51.417221+0000","mapping_epoch":13,"log_start":"48'800","ondisk_log_start":"48'800","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"113'553145","reported_seq":1103204,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:08:53.489466+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:08:53.489466+0000","last_peered":"2023-10-17T02:08:53.489466+0000","last_clean":"2023-10-17T02:08:53.489466+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:08:53.489466+0000","last_undegraded":"2023-10-17T02:08:53.489466+0000","last_fullsized":"2023-10-17T02:08:53.489466+0000","mapping_epoch":16,"log_start":"111'543110","ondisk_log_start":"111'543110","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7234109440,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":550746,"num_read_kb":2170180,"num_write":550388,"num_write_kb":2201552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":272,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:08:08.179831+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:08:08.179831+0000","last_peered":"2023-10-17T02:08:08.179831+0000","last_clean":"2023-10-17T02:08:08.179831+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:08:08.179831+0000","last_undegraded":"2023-10-17T02:08:08.179831+0000","last_fullsized":"2023-10-17T02:08:08.179831+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"113'531967","reported_seq":1061932,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:08:54.175262+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:08:54.175262+0000","last_peered":"2023-10-17T02:08:54.175262+0000","last_clean":"2023-10-17T02:08:54.175262+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:08:54.175262+0000","last_undegraded":"2023-10-17T02:08:54.175262+0000","last_fullsized":"2023-10-17T02:08:54.175262+0000","mapping_epoch":16,"log_start":"111'521919","ondisk_log_start":"111'521919","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6955347968,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":530645,"num_read_kb":2092960,"num_write":529317,"num_write_kb":2117268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2971,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:08:49.663743+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:08:49.663743+0000","last_peered":"2023-10-17T02:08:49.663743+0000","last_clean":"2023-10-17T02:08:49.663743+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:08:49.663743+0000","last_undegraded":"2023-10-17T02:08:49.663743+0000","last_fullsized":"2023-10-17T02:08:49.663743+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"113'554180","reported_seq":1105980,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:08:53.468043+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:08:53.468043+0000","last_peered":"2023-10-17T02:08:53.468043+0000","last_clean":"2023-10-17T02:08:53.468043+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:08:53.468043+0000","last_undegraded":"2023-10-17T02:08:53.468043+0000","last_fullsized":"2023-10-17T02:08:53.468043+0000","mapping_epoch":16,"log_start":"111'544125","ondisk_log_start":"111'544125","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7250915328,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":552519,"num_read_kb":2178100,"num_write":551426,"num_write_kb":2205704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4452,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:08:49.663324+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:08:49.663324+0000","last_peered":"2023-10-17T02:08:49.663324+0000","last_clean":"2023-10-17T02:08:49.663324+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:08:49.663324+0000","last_undegraded":"2023-10-17T02:08:49.663324+0000","last_fullsized":"2023-10-17T02:08:49.663324+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"113'541307","reported_seq":1082169,"reported_epoch":113,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:08:54.184294+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:08:54.184294+0000","last_peered":"2023-10-17T02:08:54.184294+0000","last_clean":"2023-10-17T02:08:54.184294+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:08:54.184294+0000","last_undegraded":"2023-10-17T02:08:54.184294+0000","last_fullsized":"2023-10-17T02:08:54.184294+0000","mapping_epoch":16,"log_start":"111'531218","ondisk_log_start":"111'531218","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1255,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 448s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7094149139,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":541493,"num_read_kb":2134136,"num_write":539337,"num_write_kb":2157342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"113'10818","reported_seq":15139,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:08:51.411991+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:08:51.411991+0000","last_peered":"2023-10-17T02:08:51.411991+0000","last_clean":"2023-10-17T02:08:51.411991+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:08:51.411991+0000","last_undegraded":"2023-10-17T02:08:51.411991+0000","last_fullsized":"2023-10-17T02:08:51.411991+0000","mapping_epoch":13,"log_start":"48'800","ondisk_log_start":"48'800","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":293,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:08:44.132595+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:08:44.132595+0000","last_peered":"2023-10-17T02:08:44.132595+0000","last_clean":"2023-10-17T02:08:44.132595+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:08:44.132595+0000","last_undegraded":"2023-10-17T02:08:44.132595+0000","last_fullsized":"2023-10-17T02:08:44.132595+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28534521875,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2175403,"num_read_kb":8575376,"num_write":2170468,"num_write_kb":8681866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17776009216,"data_stored":21176868902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40227,"ondisk_log_size":40227,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":47616,"ondisk_log_size":47616,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739412,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10379736,"kb_used_data":8677324,"kb_used_omap":832,"kb_used_meta":1701567,"kb_avail":83992104,"statfs":{"total":96636764160,"available":86007914496,"internally_reserved":0,"allocated":8885579776,"data_stored":10585368453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":852977,"internal_metadata":1742404623},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:08:29 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.52000000000000002,"15min":0.435},"min":{"1min":0.184,"5min":0.184,"15min":0.16600000000000001},"max":{"1min":1.518,"5min":12.852,"15min":12.852},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.311,"5min":0.51800000000000002,"15min":0.51200000000000001},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.192},"max":{"1min":0.51700000000000002,"5min":12.784000000000001,"15min":12.784000000000001},"last":0.312}]},{"osd":1,"last update":"Tue Oct 17 02:08:03 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52500000000000002,"15min":0.49299999999999999},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":4.6980000000000004,"5min":6.202,"15min":9.5429999999999993},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.64400000000000002,"15min":0.60499999999999998},"min":{"1min":0.27200000000000002,"5min":0.23799999999999999,"15min":0.23100000000000001},"max":{"1min":4.6550000000000002,"5min":12.824999999999999,"15min":12.824999999999999},"last":0.29599999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739409,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5684908,"kb_used_data":4275620,"kb_used_omap":364,"kb_used_meta":1408915,"kb_avail":88686932,"statfs":{"total":96636764160,"available":90815418368,"internally_reserved":0,"allocated":4378234880,"data_stored":5215797156,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":372742,"internal_metadata":1442729978},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,1,6,10,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:08:27 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.76500000000000001,"15min":0.56599999999999995},"min":{"1min":0.17699999999999999,"5min":0.16900000000000001,"15min":0.16300000000000001},"max":{"1min":4.2480000000000002,"5min":11.891999999999999,"15min":11.891999999999999},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.80900000000000005,"15min":0.61399999999999999},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.22},"max":{"1min":4.2869999999999999,"5min":11.926,"15min":11.926},"last":0.317}]},{"osd":2,"last update":"Tue Oct 17 02:08:27 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.84299999999999997,"15min":0.63400000000000001},"min":{"1min":0.20300000000000001,"5min":0.19500000000000001,"15min":0.192},"max":{"1min":2.8279999999999998,"5min":25.128,"15min":25.128},"last":1.448},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.45900000000000002,"15min":0.46500000000000002},"min":{"1min":0.214,"5min":0.214,"15min":0.20100000000000001},"max":{"1min":2.669,"5min":5.2089999999999996,"15min":6.7800000000000002},"last":0.23999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739409,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5939248,"kb_used_data":4408232,"kb_used_omap":417,"kb_used_meta":1530590,"kb_avail":88432592,"statfs":{"total":96636764160,"available":90554974208,"internally_reserved":0,"allocated":4514029568,"data_stored":5376437355,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":427766,"internal_metadata":1567324426},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,4,6,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:08:02 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.55800000000000005,"15min":0.46600000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":4.0990000000000002,"5min":6.1360000000000001,"15min":6.1360000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.67100000000000004,"15min":0.497},"min":{"1min":0.20599999999999999,"5min":0.191,"15min":0.191},"max":{"1min":5.5119999999999996,"5min":7.4880000000000004,"15min":7.4880000000000004},"last":0.19600000000000001}]},{"osd":2,"last update":"Tue Oct 17 02:08:09 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.70699999999999996,"15min":0.54100000000000004},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":5.6580000000000004,"5min":10.125,"15min":10.125},"last":0.376},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.622,"15min":0.60799999999999998},"min":{"1min":0.245,"5min":0.23699999999999999,"15min":0.20499999999999999},"max":{"1min":5.5529999999999999,"5min":6.3899999999999997,"15min":10.378},"last":0.23300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4513030144,"data_stored":5375627264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4377219072,"data_stored":5215051795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8885760000,"data_stored":10586189843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:08:57.527 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:08:57.527 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:09:17.529 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:09:18.047 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:09:18.048 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:09:18.106 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2726,"stamp":"2023-10-17T02:09:17.595127+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28566203224,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2207135,"num_read_kb":8626981,"num_write":2178708,"num_write_kb":8714483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87999,"ondisk_log_size":87999,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22266696,"kb_used_data":17429616,"kb_used_omap":1614,"kb_used_meta":4835441,"kb_avail":260848824,"statfs":{"total":289910292480,"available":267109195776,"internally_reserved":0,"allocated":17847926784,"data_stored":21247685524,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1653485,"internal_metadata":4951491859},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,5,8,13,11,13,11],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15798272,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3902,"num_read_kb":15540,"num_write":3866,"num_write_kb":15464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002046"},"pg_stats":[{"pgid":"2.7","version":"113'14390","reported_seq":20235,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:09:13.287071+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:09:13.287071+0000","last_peered":"2023-10-17T02:09:13.287071+0000","last_clean":"2023-10-17T02:09:13.287071+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:09:13.287071+0000","last_undegraded":"2023-10-17T02:09:13.287071+0000","last_fullsized":"2023-10-17T02:09:13.287071+0000","mapping_epoch":13,"log_start":"81'4300","ondisk_log_start":"81'4300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"113'10838","reported_seq":16398,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:09:13.287521+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:09:13.287521+0000","last_peered":"2023-10-17T02:09:13.287521+0000","last_clean":"2023-10-17T02:09:13.287521+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:09:13.287521+0000","last_undegraded":"2023-10-17T02:09:13.287521+0000","last_fullsized":"2023-10-17T02:09:13.287521+0000","mapping_epoch":13,"log_start":"48'800","ondisk_log_start":"48'800","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"113'7265","reported_seq":10281,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:09:13.270893+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:09:13.270893+0000","last_peered":"2023-10-17T02:09:13.270893+0000","last_clean":"2023-10-17T02:09:13.270893+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:09:13.270893+0000","last_undegraded":"2023-10-17T02:09:13.270893+0000","last_fullsized":"2023-10-17T02:09:13.270893+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":7265,"log_dups_size":0,"ondisk_log_size":7265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"113'10873","reported_seq":15842,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:09:13.270895+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:09:13.270895+0000","last_peered":"2023-10-17T02:09:13.270895+0000","last_clean":"2023-10-17T02:09:13.270895+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:09:13.270895+0000","last_undegraded":"2023-10-17T02:09:13.270895+0000","last_fullsized":"2023-10-17T02:09:13.270895+0000","mapping_epoch":13,"log_start":"48'800","ondisk_log_start":"48'800","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"113'555085","reported_seq":1107118,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:09:13.595010+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:09:13.595010+0000","last_peered":"2023-10-17T02:09:13.595010+0000","last_clean":"2023-10-17T02:09:13.595010+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:09:13.595010+0000","last_undegraded":"2023-10-17T02:09:13.595010+0000","last_fullsized":"2023-10-17T02:09:13.595010+0000","mapping_epoch":16,"log_start":"111'545010","ondisk_log_start":"111'545010","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7242080256,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":552713,"num_read_kb":2178008,"num_write":552328,"num_write_kb":2209312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":272,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:08:08.179831+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:08:08.179831+0000","last_peered":"2023-10-17T02:08:08.179831+0000","last_clean":"2023-10-17T02:08:08.179831+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:08:08.179831+0000","last_undegraded":"2023-10-17T02:08:08.179831+0000","last_fullsized":"2023-10-17T02:08:08.179831+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"113'533870","reported_seq":1065714,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:09:14.321403+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:09:14.321403+0000","last_peered":"2023-10-17T02:09:14.321403+0000","last_clean":"2023-10-17T02:09:14.321403+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:09:14.321403+0000","last_undegraded":"2023-10-17T02:09:14.321403+0000","last_fullsized":"2023-10-17T02:09:14.321403+0000","mapping_epoch":16,"log_start":"111'523819","ondisk_log_start":"111'523819","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6963068928,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":532522,"num_read_kb":2100444,"num_write":531220,"num_write_kb":2124880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2981,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:09:15.264374+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:09:15.264374+0000","last_peered":"2023-10-17T02:09:15.264374+0000","last_clean":"2023-10-17T02:09:15.264374+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:09:15.264374+0000","last_undegraded":"2023-10-17T02:09:15.264374+0000","last_fullsized":"2023-10-17T02:09:15.264374+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"113'556049","reported_seq":1109856,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:09:13.555858+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:09:13.555858+0000","last_peered":"2023-10-17T02:09:13.555858+0000","last_clean":"2023-10-17T02:09:13.555858+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:09:13.555858+0000","last_undegraded":"2023-10-17T02:09:13.555858+0000","last_fullsized":"2023-10-17T02:09:13.555858+0000","mapping_epoch":16,"log_start":"111'546025","ondisk_log_start":"111'546025","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7258550272,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":554524,"num_read_kb":2186104,"num_write":553295,"num_write_kb":2213180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4467,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:09:15.263539+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:09:15.263539+0000","last_peered":"2023-10-17T02:09:15.263539+0000","last_clean":"2023-10-17T02:09:15.263539+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:09:15.263539+0000","last_undegraded":"2023-10-17T02:09:15.263539+0000","last_fullsized":"2023-10-17T02:09:15.263539+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"113'543232","reported_seq":1086021,"reported_epoch":113,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:09:14.329842+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:09:14.329842+0000","last_peered":"2023-10-17T02:09:14.329842+0000","last_clean":"2023-10-17T02:09:14.329842+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:09:14.329842+0000","last_undegraded":"2023-10-17T02:09:14.329842+0000","last_fullsized":"2023-10-17T02:09:14.329842+0000","mapping_epoch":16,"log_start":"111'533218","ondisk_log_start":"111'533218","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1307,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 468s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7101911059,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":543414,"num_read_kb":2141772,"num_write":541235,"num_write_kb":2164934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"113'10842","reported_seq":15175,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:09:13.318862+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:09:13.318862+0000","last_peered":"2023-10-17T02:09:13.318862+0000","last_clean":"2023-10-17T02:09:13.318862+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:09:13.318862+0000","last_undegraded":"2023-10-17T02:09:13.318862+0000","last_fullsized":"2023-10-17T02:09:13.318862+0000","mapping_epoch":13,"log_start":"48'800","ondisk_log_start":"48'800","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":293,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:08:44.132595+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:08:44.132595+0000","last_peered":"2023-10-17T02:08:44.132595+0000","last_clean":"2023-10-17T02:08:44.132595+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:08:44.132595+0000","last_undegraded":"2023-10-17T02:08:44.132595+0000","last_fullsized":"2023-10-17T02:08:44.132595+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28565610515,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2183173,"num_read_kb":8606328,"num_write":2178078,"num_write_kb":8712306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17846136832,"data_stored":21246996518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40164,"ondisk_log_size":40164,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":47736,"ondisk_log_size":47736,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739417,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10544068,"kb_used_data":8715324,"kb_used_omap":832,"kb_used_meta":1827903,"kb_avail":83827772,"statfs":{"total":96636764160,"available":85839638528,"internally_reserved":0,"allocated":8924491776,"data_stored":10624280453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":852977,"internal_metadata":1871772687},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,5,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:08:29 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.52000000000000002,"15min":0.435},"min":{"1min":0.184,"5min":0.184,"15min":0.16600000000000001},"max":{"1min":1.518,"5min":12.852,"15min":12.852},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.311,"5min":0.51800000000000002,"15min":0.51200000000000001},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.192},"max":{"1min":0.51700000000000002,"5min":12.784000000000001,"15min":12.784000000000001},"last":0.28799999999999998}]},{"osd":1,"last update":"Tue Oct 17 02:09:05 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.45000000000000001,"15min":0.46800000000000003},"min":{"1min":0.246,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.76100000000000001,"5min":4.6980000000000004,"15min":9.5429999999999993},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.61899999999999999,"15min":0.60599999999999998},"min":{"1min":0.253,"5min":0.23799999999999999,"15min":0.23100000000000001},"max":{"1min":4.2539999999999996,"5min":12.824999999999999,"15min":12.824999999999999},"last":0.35099999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739413,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5736292,"kb_used_data":4290908,"kb_used_omap":364,"kb_used_meta":1445011,"kb_avail":88635548,"statfs":{"total":96636764160,"available":90762801152,"internally_reserved":0,"allocated":4393889792,"data_stored":5231452068,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":372742,"internal_metadata":1479692282},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,4,1,11,11],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:08:27 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.76500000000000001,"15min":0.56599999999999995},"min":{"1min":0.17699999999999999,"5min":0.16900000000000001,"15min":0.16300000000000001},"max":{"1min":4.2480000000000002,"5min":11.891999999999999,"15min":11.891999999999999},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.80900000000000005,"15min":0.61399999999999999},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.22},"max":{"1min":4.2869999999999999,"5min":11.926,"15min":11.926},"last":0.32600000000000001}]},{"osd":2,"last update":"Tue Oct 17 02:08:27 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.84299999999999997,"15min":0.63400000000000001},"min":{"1min":0.20300000000000001,"5min":0.19500000000000001,"15min":0.192},"max":{"1min":2.8279999999999998,"5min":25.128,"15min":25.128},"last":0.373},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.45900000000000002,"15min":0.46500000000000002},"min":{"1min":0.214,"5min":0.214,"15min":0.20100000000000001},"max":{"1min":2.669,"5min":5.2089999999999996,"15min":6.7800000000000002},"last":0.25}]}]},{"osd":0,"up_from":8,"seq":34359739413,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5986336,"kb_used_data":4423384,"kb_used_omap":417,"kb_used_meta":1562526,"kb_avail":88385504,"statfs":{"total":96636764160,"available":90506756096,"internally_reserved":0,"allocated":4529545216,"data_stored":5391953003,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":427766,"internal_metadata":1600026890},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,4,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:09:06 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.60699999999999998,"15min":0.48999999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":5.4370000000000003,"5min":6.1360000000000001,"15min":6.1360000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.64100000000000001,"15min":0.48599999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.191},"max":{"1min":4.0880000000000001,"5min":7.4880000000000004,"15min":7.4880000000000004},"last":0.32300000000000001}]},{"osd":2,"last update":"Tue Oct 17 02:08:09 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.70699999999999996,"15min":0.54100000000000004},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":5.6580000000000004,"5min":10.125,"15min":10.125},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.622,"15min":0.60799999999999998},"min":{"1min":0.245,"5min":0.23699999999999999,"15min":0.20499999999999999},"max":{"1min":5.5529999999999999,"5min":6.3899999999999997,"15min":10.378},"last":0.34899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4528640000,"data_stored":5391237120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4392775680,"data_stored":5230608403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8924721152,"data_stored":10625150995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:09:18.108 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:09:18.109 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-17T02:09:18.647 INFO:teuthology.orchestra.run.smithi169.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-17T02:09:18.689 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:09:35.679 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:09:35.678+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T02:09:35.705 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:09:35.698+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T02:09:35.706 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:09:35.698+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T02:09:35.706 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:09:35.702+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T02:09:35.710 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:09:35.706+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T02:09:35.724 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:09:35.706+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T02:09:38.690 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:09:39.318 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:09:39.319 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:09:39.348 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2736,"stamp":"2023-10-17T02:09:37.601827+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28596874072,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2214535,"num_read_kb":8656485,"num_write":2186195,"num_write_kb":8744431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88045,"ondisk_log_size":88045,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22581952,"kb_used_data":17490344,"kb_used_omap":1614,"kb_used_meta":5089969,"kb_avail":260533568,"statfs":{"total":289910292480,"available":266786373632,"internally_reserved":0,"allocated":17910112256,"data_stored":21309870996,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1653485,"internal_metadata":5212128531},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,3,2,9,17,18,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15728640,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3718,"num_read_kb":14820,"num_write":3817,"num_write_kb":15268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005425"},"pg_stats":[{"pgid":"2.7","version":"113'14450","reported_seq":20312,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:09:34.467634+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:09:34.467634+0000","last_peered":"2023-10-17T02:09:34.467634+0000","last_clean":"2023-10-17T02:09:34.467634+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:09:34.467634+0000","last_undegraded":"2023-10-17T02:09:34.467634+0000","last_fullsized":"2023-10-17T02:09:34.467634+0000","mapping_epoch":13,"log_start":"81'4400","ondisk_log_start":"81'4400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"113'10883","reported_seq":16455,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:09:34.459449+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:09:34.459449+0000","last_peered":"2023-10-17T02:09:34.459449+0000","last_clean":"2023-10-17T02:09:34.459449+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:09:34.459449+0000","last_undegraded":"2023-10-17T02:09:34.459449+0000","last_fullsized":"2023-10-17T02:09:34.459449+0000","mapping_epoch":13,"log_start":"48'800","ondisk_log_start":"48'800","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"113'7295","reported_seq":10319,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:09:34.447133+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:09:34.447133+0000","last_peered":"2023-10-17T02:09:34.447133+0000","last_clean":"2023-10-17T02:09:34.447133+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:09:34.447133+0000","last_undegraded":"2023-10-17T02:09:34.447133+0000","last_fullsized":"2023-10-17T02:09:34.447133+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":7295,"log_dups_size":0,"ondisk_log_size":7295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"113'10918","reported_seq":15899,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:09:34.454711+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:09:34.454711+0000","last_peered":"2023-10-17T02:09:34.454711+0000","last_clean":"2023-10-17T02:09:34.454711+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:09:34.454711+0000","last_undegraded":"2023-10-17T02:09:34.454711+0000","last_fullsized":"2023-10-17T02:09:34.454711+0000","mapping_epoch":13,"log_start":"50'900","ondisk_log_start":"50'900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"113'556946","reported_seq":1110831,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:09:33.673270+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:09:33.673270+0000","last_peered":"2023-10-17T02:09:33.673270+0000","last_clean":"2023-10-17T02:09:33.673270+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:09:33.673270+0000","last_undegraded":"2023-10-17T02:09:33.673270+0000","last_fullsized":"2023-10-17T02:09:33.673270+0000","mapping_epoch":16,"log_start":"112'546910","ondisk_log_start":"112'546910","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7249735680,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":554566,"num_read_kb":2185400,"num_write":554189,"num_write_kb":2216756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":272,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:08:08.179831+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:08:08.179831+0000","last_peered":"2023-10-17T02:08:08.179831+0000","last_clean":"2023-10-17T02:08:08.179831+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:08:08.179831+0000","last_undegraded":"2023-10-17T02:08:08.179831+0000","last_fullsized":"2023-10-17T02:08:08.179831+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"113'535789","reported_seq":1069428,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:09:34.450482+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:09:34.450482+0000","last_peered":"2023-10-17T02:09:34.450482+0000","last_clean":"2023-10-17T02:09:34.450482+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:09:34.450482+0000","last_undegraded":"2023-10-17T02:09:34.450482+0000","last_fullsized":"2023-10-17T02:09:34.450482+0000","mapping_epoch":16,"log_start":"112'525719","ondisk_log_start":"112'525719","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6970945536,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":534317,"num_read_kb":2107616,"num_write":533139,"num_write_kb":2132556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2989,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:09:35.768674+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:09:35.768674+0000","last_peered":"2023-10-17T02:09:35.768674+0000","last_clean":"2023-10-17T02:09:35.768674+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:09:35.768674+0000","last_undegraded":"2023-10-17T02:09:35.768674+0000","last_fullsized":"2023-10-17T02:09:35.768674+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"113'557888","reported_seq":1113603,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:09:33.685747+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:09:33.685747+0000","last_peered":"2023-10-17T02:09:33.685747+0000","last_clean":"2023-10-17T02:09:33.685747+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:09:33.685747+0000","last_undegraded":"2023-10-17T02:09:33.685747+0000","last_fullsized":"2023-10-17T02:09:33.685747+0000","mapping_epoch":16,"log_start":"112'547825","ondisk_log_start":"112'547825","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7266054144,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":556441,"num_read_kb":2193728,"num_write":555134,"num_write_kb":2220536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4479,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:09:35.752236+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:09:35.752236+0000","last_peered":"2023-10-17T02:09:35.752236+0000","last_clean":"2023-10-17T02:09:35.752236+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:09:35.752236+0000","last_undegraded":"2023-10-17T02:09:35.752236+0000","last_fullsized":"2023-10-17T02:09:35.752236+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"113'545134","reported_seq":1089771,"reported_epoch":113,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:09:34.454642+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:09:34.454642+0000","last_peered":"2023-10-17T02:09:34.454642+0000","last_clean":"2023-10-17T02:09:34.454642+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:09:34.454642+0000","last_undegraded":"2023-10-17T02:09:34.454642+0000","last_fullsized":"2023-10-17T02:09:34.454642+0000","mapping_epoch":16,"log_start":"112'535118","ondisk_log_start":"112'535118","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1367,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 489s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7109546003,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":545249,"num_read_kb":2149088,"num_write":543103,"num_write_kb":2172406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"113'10887","reported_seq":15230,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:09:33.742950+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:09:33.742950+0000","last_peered":"2023-10-17T02:09:33.742950+0000","last_clean":"2023-10-17T02:09:33.742950+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:09:33.742950+0000","last_undegraded":"2023-10-17T02:09:33.742950+0000","last_fullsized":"2023-10-17T02:09:33.742950+0000","mapping_epoch":13,"log_start":"48'800","ondisk_log_start":"48'800","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":293,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:08:44.132595+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:08:44.132595+0000","last_peered":"2023-10-17T02:08:44.132595+0000","last_clean":"2023-10-17T02:08:44.132595+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:08:44.132595+0000","last_undegraded":"2023-10-17T02:08:44.132595+0000","last_fullsized":"2023-10-17T02:08:44.132595+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28596281363,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2190573,"num_read_kb":8635832,"num_write":2185565,"num_write_kb":8742254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17907425280,"data_stored":21308284966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40185,"ondisk_log_size":40185,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":47761,"ondisk_log_size":47761,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739421,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10695684,"kb_used_data":8746172,"kb_used_omap":832,"kb_used_meta":1948671,"kb_avail":83676156,"statfs":{"total":96636764160,"available":85684383744,"internally_reserved":0,"allocated":8956080128,"data_stored":10655868805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":852977,"internal_metadata":1995439119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:09:30 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.51200000000000001,"15min":0.44500000000000001},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":2.0329999999999999,"5min":12.852,"15min":12.852},"last":0.245},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.504,"15min":0.51900000000000002},"min":{"1min":0.22500000000000001,"5min":0.20499999999999999,"15min":0.192},"max":{"1min":1.738,"5min":12.784000000000001,"15min":12.784000000000001},"last":0.27700000000000002}]},{"osd":1,"last update":"Tue Oct 17 02:09:05 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.45000000000000001,"15min":0.46800000000000003},"min":{"1min":0.246,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.76100000000000001,"5min":4.6980000000000004,"15min":9.5429999999999993},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.61899999999999999,"15min":0.60599999999999998},"min":{"1min":0.253,"5min":0.23799999999999999,"15min":0.23100000000000001},"max":{"1min":4.2539999999999996,"5min":12.824999999999999,"15min":12.824999999999999},"last":0.33100000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739417,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5831044,"kb_used_data":4306044,"kb_used_omap":364,"kb_used_meta":1524627,"kb_avail":88540796,"statfs":{"total":96636764160,"available":90665775104,"internally_reserved":0,"allocated":4409389056,"data_stored":5246951332,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":372742,"internal_metadata":1561219066},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,6,7,14,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:09:29 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.81000000000000005,"15min":0.58899999999999997},"min":{"1min":0.17100000000000001,"5min":0.16900000000000001,"15min":0.16300000000000001},"max":{"1min":3.347,"5min":11.891999999999999,"15min":11.891999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.747,"5min":0.85499999999999998,"15min":0.63800000000000001},"min":{"1min":0.224,"5min":0.224,"15min":0.22},"max":{"1min":3.4119999999999999,"5min":11.926,"15min":11.926},"last":0.46000000000000002}]},{"osd":2,"last update":"Tue Oct 17 02:09:29 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.879,"15min":0.66800000000000004},"min":{"1min":0.20999999999999999,"5min":0.20300000000000001,"15min":0.192},"max":{"1min":5.5490000000000004,"5min":25.128,"15min":25.128},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.47599999999999998,"15min":0.48399999999999999},"min":{"1min":0.22900000000000001,"5min":0.214,"15min":0.20100000000000001},"max":{"1min":5.2709999999999999,"5min":5.2709999999999999,"15min":6.7800000000000002},"last":0.25900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739417,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6055224,"kb_used_data":4438128,"kb_used_omap":417,"kb_used_meta":1616670,"kb_avail":88316616,"statfs":{"total":96636764160,"available":90436214784,"internally_reserved":0,"allocated":4544643072,"data_stored":5407050859,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":427766,"internal_metadata":1655470346},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,0,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:09:06 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.60699999999999998,"15min":0.48999999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":5.4370000000000003,"5min":6.1360000000000001,"15min":6.1360000000000001},"last":0.19900000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.64100000000000001,"15min":0.48599999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.191},"max":{"1min":4.0880000000000001,"5min":7.4880000000000004,"15min":7.4880000000000004},"last":0.23699999999999999}]},{"osd":2,"last update":"Tue Oct 17 02:09:14 2023","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":0.85199999999999998,"15min":0.59299999999999997},"min":{"1min":0.23499999999999999,"5min":0.187,"15min":0.187},"max":{"1min":12.382,"5min":12.382,"15min":12.382},"last":0.29899999999999999},{"interface":"front","average":{"1min":1.0349999999999999,"5min":0.68899999999999995,"15min":0.623},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.20499999999999999},"max":{"1min":12.561,"5min":12.561,"15min":12.561},"last":0.28399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4543787008,"data_stored":5406384128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4408283136,"data_stored":5246115859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8955355136,"data_stored":10655784979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:09:39.350 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:09:39.350 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:09:59.352 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:09:59.806 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:09:59.807 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:09:59.854 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2748,"stamp":"2023-10-17T02:09:59.605434+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28631685976,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2222906,"num_read_kb":8689857,"num_write":2194630,"num_write_kb":8778171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88017,"ondisk_log_size":88017,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22887944,"kb_used_data":17554284,"kb_used_omap":1614,"kb_used_meta":5332017,"kb_avail":260227576,"statfs":{"total":289910292480,"available":266473037824,"internally_reserved":0,"allocated":17975586816,"data_stored":21375340940,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1653485,"internal_metadata":5459985683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,10,15,21,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11530240,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2744,"num_read_kb":10936,"num_write":2803,"num_write_kb":11212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001690"},"pg_stats":[{"pgid":"2.7","version":"113'14507","reported_seq":20385,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:09:54.544704+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:09:54.544704+0000","last_peered":"2023-10-17T02:09:54.544704+0000","last_clean":"2023-10-17T02:09:54.544704+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:09:54.544704+0000","last_undegraded":"2023-10-17T02:09:54.544704+0000","last_fullsized":"2023-10-17T02:09:54.544704+0000","mapping_epoch":13,"log_start":"83'4500","ondisk_log_start":"83'4500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"113'10926","reported_seq":16510,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:09:54.540245+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:09:54.540245+0000","last_peered":"2023-10-17T02:09:54.540245+0000","last_clean":"2023-10-17T02:09:54.540245+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:09:54.540245+0000","last_undegraded":"2023-10-17T02:09:54.540245+0000","last_fullsized":"2023-10-17T02:09:54.540245+0000","mapping_epoch":13,"log_start":"50'900","ondisk_log_start":"50'900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"113'7324","reported_seq":10356,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:09:54.522807+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:09:54.522807+0000","last_peered":"2023-10-17T02:09:54.522807+0000","last_clean":"2023-10-17T02:09:54.522807+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:09:54.522807+0000","last_undegraded":"2023-10-17T02:09:54.522807+0000","last_fullsized":"2023-10-17T02:09:54.522807+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":7324,"log_dups_size":0,"ondisk_log_size":7324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"113'10961","reported_seq":15954,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:09:54.528900+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:09:54.528900+0000","last_peered":"2023-10-17T02:09:54.528900+0000","last_clean":"2023-10-17T02:09:54.528900+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:09:54.528900+0000","last_undegraded":"2023-10-17T02:09:54.528900+0000","last_fullsized":"2023-10-17T02:09:54.528900+0000","mapping_epoch":13,"log_start":"50'900","ondisk_log_start":"50'900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"114'559303","reported_seq":1115476,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:09:58.870125+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:09:58.870125+0000","last_peered":"2023-10-17T02:09:58.870125+0000","last_clean":"2023-10-17T02:09:58.870125+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:09:58.870125+0000","last_undegraded":"2023-10-17T02:09:58.870125+0000","last_fullsized":"2023-10-17T02:09:58.870125+0000","mapping_epoch":16,"log_start":"112'549210","ondisk_log_start":"112'549210","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7259394048,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":556853,"num_read_kb":2194520,"num_write":556546,"num_write_kb":2226184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":273,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:08:47.471489+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:08:47.471489+0000","last_peered":"2023-10-17T02:08:47.471489+0000","last_clean":"2023-10-17T02:08:47.471489+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:08:47.471489+0000","last_undegraded":"2023-10-17T02:08:47.471489+0000","last_fullsized":"2023-10-17T02:08:47.471489+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"113'537623","reported_seq":1073098,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:09:54.560143+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:09:54.560143+0000","last_peered":"2023-10-17T02:09:54.560143+0000","last_clean":"2023-10-17T02:09:54.560143+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:09:54.560143+0000","last_undegraded":"2023-10-17T02:09:54.560143+0000","last_fullsized":"2023-10-17T02:09:54.560143+0000","mapping_epoch":16,"log_start":"112'527619","ondisk_log_start":"112'527619","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6978506752,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":536160,"num_read_kb":2114972,"num_write":534973,"num_write_kb":2139892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":2997,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:09:56.232460+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:09:56.232460+0000","last_peered":"2023-10-17T02:09:56.232460+0000","last_clean":"2023-10-17T02:09:56.232460+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:09:56.232460+0000","last_undegraded":"2023-10-17T02:09:56.232460+0000","last_fullsized":"2023-10-17T02:09:56.232460+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"114'560300","reported_seq":1118386,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:09:58.863988+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:09:58.863988+0000","last_peered":"2023-10-17T02:09:58.863988+0000","last_clean":"2023-10-17T02:09:58.863988+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:09:58.863988+0000","last_undegraded":"2023-10-17T02:09:58.863988+0000","last_fullsized":"2023-10-17T02:09:58.863988+0000","mapping_epoch":16,"log_start":"112'550225","ondisk_log_start":"112'550225","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7276011520,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":558799,"num_read_kb":2203124,"num_write":557546,"num_write_kb":2230184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4491,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:09:56.226185+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:09:56.226185+0000","last_peered":"2023-10-17T02:09:56.226185+0000","last_clean":"2023-10-17T02:09:56.226185+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:09:56.226185+0000","last_undegraded":"2023-10-17T02:09:56.226185+0000","last_fullsized":"2023-10-17T02:09:56.226185+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"113'546987","reported_seq":1093515,"reported_epoch":113,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:09:54.549557+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:09:54.549557+0000","last_peered":"2023-10-17T02:09:54.549557+0000","last_clean":"2023-10-17T02:09:54.549557+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:09:54.549557+0000","last_undegraded":"2023-10-17T02:09:54.549557+0000","last_fullsized":"2023-10-17T02:09:54.549557+0000","mapping_epoch":16,"log_start":"112'536918","ondisk_log_start":"112'536918","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1409,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 509s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7117180947,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":547132,"num_read_kb":2156588,"num_write":544935,"num_write_kb":2179734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"113'10931","reported_seq":15288,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:09:54.555898+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:09:54.555898+0000","last_peered":"2023-10-17T02:09:54.555898+0000","last_clean":"2023-10-17T02:09:54.555898+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:09:54.555898+0000","last_undegraded":"2023-10-17T02:09:54.555898+0000","last_fullsized":"2023-10-17T02:09:54.555898+0000","mapping_epoch":13,"log_start":"50'900","ondisk_log_start":"50'900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":293,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-17T02:08:44.132595+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:08:44.132595+0000","last_peered":"2023-10-17T02:08:44.132595+0000","last_clean":"2023-10-17T02:08:44.132595+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:08:44.132595+0000","last_undegraded":"2023-10-17T02:08:44.132595+0000","last_fullsized":"2023-10-17T02:08:44.132595+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28631093267,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2198944,"num_read_kb":8669204,"num_write":2194000,"num_write_kb":8775994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17972928512,"data_stored":21373788198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40241,"ondisk_log_size":40241,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":47677,"ondisk_log_size":47677,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739425,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10855616,"kb_used_data":8776440,"kb_used_omap":832,"kb_used_meta":2078335,"kb_avail":83516224,"statfs":{"total":96636764160,"available":85520613376,"internally_reserved":0,"allocated":8987074560,"data_stored":10686863237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":852977,"internal_metadata":2128215055},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:09:30 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.51200000000000001,"15min":0.44500000000000001},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":2.0329999999999999,"5min":12.852,"15min":12.852},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.504,"15min":0.51900000000000002},"min":{"1min":0.22500000000000001,"5min":0.20499999999999999,"15min":0.192},"max":{"1min":1.738,"5min":12.784000000000001,"15min":12.784000000000001},"last":0.51700000000000002}]},{"osd":1,"last update":"Tue Oct 17 02:09:05 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.45000000000000001,"15min":0.46800000000000003},"min":{"1min":0.246,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.76100000000000001,"5min":4.6980000000000004,"15min":9.5429999999999993},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.61899999999999999,"15min":0.60599999999999998},"min":{"1min":0.253,"5min":0.23799999999999999,"15min":0.23100000000000001},"max":{"1min":4.2539999999999996,"5min":12.824999999999999,"15min":12.824999999999999},"last":0.442}]}]},{"osd":1,"up_from":8,"seq":34359739421,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5899796,"kb_used_data":4320716,"kb_used_omap":364,"kb_used_meta":1578707,"kb_avail":88472044,"statfs":{"total":96636764160,"available":90595373056,"internally_reserved":0,"allocated":4424413184,"data_stored":5261975460,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":372742,"internal_metadata":1616596986},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,5,12,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:09:29 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.81000000000000005,"15min":0.58899999999999997},"min":{"1min":0.17100000000000001,"5min":0.16900000000000001,"15min":0.16300000000000001},"max":{"1min":3.347,"5min":11.891999999999999,"15min":11.891999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.747,"5min":0.85499999999999998,"15min":0.63800000000000001},"min":{"1min":0.224,"5min":0.224,"15min":0.22},"max":{"1min":3.4119999999999999,"5min":11.926,"15min":11.926},"last":0.56799999999999995}]},{"osd":2,"last update":"Tue Oct 17 02:09:29 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.879,"15min":0.66800000000000004},"min":{"1min":0.20999999999999999,"5min":0.20300000000000001,"15min":0.192},"max":{"1min":5.5490000000000004,"5min":25.128,"15min":25.128},"last":0.754},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.47599999999999998,"15min":0.48399999999999999},"min":{"1min":0.22900000000000001,"5min":0.214,"15min":0.20100000000000001},"max":{"1min":5.2709999999999999,"5min":5.2709999999999999,"15min":6.7800000000000002},"last":0.34300000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739422,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6132532,"kb_used_data":4457128,"kb_used_omap":417,"kb_used_meta":1674974,"kb_avail":88239308,"statfs":{"total":96636764160,"available":90357051392,"internally_reserved":0,"allocated":4564099072,"data_stored":5426502243,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":427766,"internal_metadata":1715173642},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,8,8,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:09:06 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.60699999999999998,"15min":0.48999999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":5.4370000000000003,"5min":6.1360000000000001,"15min":6.1360000000000001},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.64100000000000001,"15min":0.48599999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.191},"max":{"1min":4.0880000000000001,"5min":7.4880000000000004,"15min":7.4880000000000004},"last":0.307}]},{"osd":2,"last update":"Tue Oct 17 02:09:14 2023","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":0.85199999999999998,"15min":0.59299999999999997},"min":{"1min":0.23499999999999999,"5min":0.187,"15min":0.187},"max":{"1min":12.382,"5min":12.382,"15min":12.382},"last":0.47199999999999998},{"interface":"front","average":{"1min":1.0349999999999999,"5min":0.68899999999999995,"15min":0.623},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.20499999999999999},"max":{"1min":12.561,"5min":12.561,"15min":12.561},"last":0.40000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4563324928,"data_stored":5425922048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4423307264,"data_stored":5261139987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8986296320,"data_stored":10686726163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:09:59.856 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:09:59.857 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:10:13.278 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:10:13.273+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T02:10:13.283 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:10:13.281+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T02:10:13.292 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:10:13.289+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T02:10:13.294 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:10:13.289+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T02:10:13.294 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:10:13.289+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T02:10:13.294 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:10:13.289+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T02:10:13.295 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:10:13.289+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T02:10:13.295 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:10:13.289+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T02:10:13.295 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:10:13.289+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T02:10:13.296 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:10:13.289+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T02:10:13.296 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:10:13.289+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T02:10:13.296 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:10:13.289+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T02:10:13.300 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:10:13.297+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T02:10:19.861 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:10:20.374 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:10:20.375 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:10:20.399 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2758,"stamp":"2023-10-17T02:10:19.609399+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28661824344,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2230234,"num_read_kb":8719073,"num_write":2201938,"num_write_kb":8807403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88155,"ondisk_log_size":88155,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22921068,"kb_used_data":17612300,"kb_used_omap":1614,"kb_used_meta":5307121,"kb_avail":260194452,"statfs":{"total":289910292480,"available":266439118848,"internally_reserved":0,"allocated":18034995200,"data_stored":21434740092,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1653485,"internal_metadata":5434492179},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,2,9,13,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19173376,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4725,"num_read_kb":18844,"num_write":4637,"num_write_kb":18548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002030"},"pg_stats":[{"pgid":"2.7","version":"114'14558","reported_seq":20453,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:14.169873+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:10:14.169873+0000","last_peered":"2023-10-17T02:10:14.169873+0000","last_clean":"2023-10-17T02:10:14.169873+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:10:14.169873+0000","last_undegraded":"2023-10-17T02:10:14.169873+0000","last_fullsized":"2023-10-17T02:10:14.169873+0000","mapping_epoch":13,"log_start":"83'4500","ondisk_log_start":"83'4500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"114'10965","reported_seq":16562,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:14.163298+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:10:14.163298+0000","last_peered":"2023-10-17T02:10:14.163298+0000","last_clean":"2023-10-17T02:10:14.163298+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:10:14.163298+0000","last_undegraded":"2023-10-17T02:10:14.163298+0000","last_fullsized":"2023-10-17T02:10:14.163298+0000","mapping_epoch":13,"log_start":"50'900","ondisk_log_start":"50'900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"114'7350","reported_seq":10392,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:14.163439+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:10:14.163439+0000","last_peered":"2023-10-17T02:10:14.163439+0000","last_clean":"2023-10-17T02:10:14.163439+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:10:14.163439+0000","last_undegraded":"2023-10-17T02:10:14.163439+0000","last_fullsized":"2023-10-17T02:10:14.163439+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":7350,"log_dups_size":0,"ondisk_log_size":7350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"114'11000","reported_seq":16007,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:14.169804+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:10:14.169804+0000","last_peered":"2023-10-17T02:10:14.169804+0000","last_clean":"2023-10-17T02:10:14.169804+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:10:14.169804+0000","last_undegraded":"2023-10-17T02:10:14.169804+0000","last_fullsized":"2023-10-17T02:10:14.169804+0000","mapping_epoch":13,"log_start":"50'900","ondisk_log_start":"50'900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"114'561205","reported_seq":1119236,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:18.974853+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:10:18.974853+0000","last_peered":"2023-10-17T02:10:18.974853+0000","last_clean":"2023-10-17T02:10:18.974853+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:10:18.974853+0000","last_undegraded":"2023-10-17T02:10:18.974853+0000","last_fullsized":"2023-10-17T02:10:18.974853+0000","mapping_epoch":16,"log_start":"112'551110","ondisk_log_start":"112'551110","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7267110912,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":558712,"num_read_kb":2201936,"num_write":558448,"num_write_kb":2233792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":274,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:09:57.471837+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:09:57.471837+0000","last_peered":"2023-10-17T02:09:57.471837+0000","last_clean":"2023-10-17T02:09:57.471837+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:09:57.471837+0000","last_undegraded":"2023-10-17T02:09:57.471837+0000","last_fullsized":"2023-10-17T02:09:57.471837+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"114'539386","reported_seq":1076616,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:14.753414+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:10:14.753414+0000","last_peered":"2023-10-17T02:10:14.753414+0000","last_clean":"2023-10-17T02:10:14.753414+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:10:14.753414+0000","last_undegraded":"2023-10-17T02:10:14.753414+0000","last_fullsized":"2023-10-17T02:10:14.753414+0000","mapping_epoch":16,"log_start":"112'529319","ondisk_log_start":"112'529319","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6985928704,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":537908,"num_read_kb":2121928,"num_write":536736,"num_write_kb":2146944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":3005,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:11.584099+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:10:11.584099+0000","last_peered":"2023-10-17T02:10:11.584099+0000","last_clean":"2023-10-17T02:10:11.584099+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:10:11.584099+0000","last_undegraded":"2023-10-17T02:10:11.584099+0000","last_fullsized":"2023-10-17T02:10:11.584099+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"114'562141","reported_seq":1122139,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:18.968787+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:10:18.968787+0000","last_peered":"2023-10-17T02:10:18.968787+0000","last_clean":"2023-10-17T02:10:18.968787+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:10:18.968787+0000","last_undegraded":"2023-10-17T02:10:18.968787+0000","last_fullsized":"2023-10-17T02:10:18.968787+0000","mapping_epoch":16,"log_start":"112'552125","ondisk_log_start":"112'552125","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7283658752,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":560712,"num_read_kb":2210752,"num_write":559387,"num_write_kb":2237548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4501,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:11.636213+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:10:11.636213+0000","last_peered":"2023-10-17T02:10:11.636213+0000","last_clean":"2023-10-17T02:10:11.636213+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:10:11.636213+0000","last_undegraded":"2023-10-17T02:10:11.636213+0000","last_fullsized":"2023-10-17T02:10:11.636213+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"114'548826","reported_seq":1097173,"reported_epoch":114,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:10:14.767654+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:10:14.767654+0000","last_peered":"2023-10-17T02:10:14.767654+0000","last_clean":"2023-10-17T02:10:14.767654+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:10:14.767654+0000","last_undegraded":"2023-10-17T02:10:14.767654+0000","last_fullsized":"2023-10-17T02:10:14.767654+0000","mapping_epoch":16,"log_start":"112'538818","ondisk_log_start":"112'538818","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1477,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 529s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7124533267,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":548940,"num_read_kb":2163804,"num_write":546737,"num_write_kb":2186942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"114'10969","reported_seq":15340,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:14.183736+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:10:14.183736+0000","last_peered":"2023-10-17T02:10:14.183736+0000","last_clean":"2023-10-17T02:10:14.183736+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:10:14.183736+0000","last_undegraded":"2023-10-17T02:10:14.183736+0000","last_fullsized":"2023-10-17T02:10:14.183736+0000","mapping_epoch":13,"log_start":"50'900","ondisk_log_start":"50'900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":294,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:08:47.455867+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:08:47.455867+0000","last_peered":"2023-10-17T02:08:47.455867+0000","last_clean":"2023-10-17T02:08:47.455867+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:08:47.455867+0000","last_undegraded":"2023-10-17T02:08:47.455867+0000","last_fullsized":"2023-10-17T02:08:47.455867+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28661231635,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2206272,"num_read_kb":8698420,"num_write":2201308,"num_write_kb":8805226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18032652288,"data_stored":21433511974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40186,"ondisk_log_size":40186,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":47870,"ondisk_log_size":47870,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739429,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10758788,"kb_used_data":8805112,"kb_used_omap":832,"kb_used_meta":1952831,"kb_avail":83613052,"statfs":{"total":96636764160,"available":85619765248,"internally_reserved":0,"allocated":9016434688,"data_stored":10716218749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":852977,"internal_metadata":1999698959},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:09:30 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.51200000000000001,"15min":0.44500000000000001},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":2.0329999999999999,"5min":12.852,"15min":12.852},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.504,"15min":0.51900000000000002},"min":{"1min":0.22500000000000001,"5min":0.20499999999999999,"15min":0.192},"max":{"1min":1.738,"5min":12.784000000000001,"15min":12.784000000000001},"last":0.27300000000000002}]},{"osd":1,"last update":"Tue Oct 17 02:10:06 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.47999999999999998,"15min":0.48499999999999999},"min":{"1min":0.224,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":4.0730000000000004,"5min":4.6980000000000004,"15min":9.5429999999999993},"last":0.223},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.52300000000000002,"15min":0.60699999999999998},"min":{"1min":0.23999999999999999,"5min":0.23799999999999999,"15min":0.23100000000000001},"max":{"1min":5.1779999999999999,"5min":5.1779999999999999,"15min":12.824999999999999},"last":0.255}]}]},{"osd":1,"up_from":8,"seq":34359739425,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5971176,"kb_used_data":4335008,"kb_used_omap":364,"kb_used_meta":1635795,"kb_avail":88400664,"statfs":{"total":96636764160,"available":90522279936,"internally_reserved":0,"allocated":4439048192,"data_stored":5276605852,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":372742,"internal_metadata":1675055098},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:09:29 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.81000000000000005,"15min":0.58899999999999997},"min":{"1min":0.17100000000000001,"5min":0.16900000000000001,"15min":0.16300000000000001},"max":{"1min":3.347,"5min":11.891999999999999,"15min":11.891999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.747,"5min":0.85499999999999998,"15min":0.63800000000000001},"min":{"1min":0.224,"5min":0.224,"15min":0.22},"max":{"1min":3.4119999999999999,"5min":11.926,"15min":11.926},"last":0.498}]},{"osd":2,"last update":"Tue Oct 17 02:09:29 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.879,"15min":0.66800000000000004},"min":{"1min":0.20999999999999999,"5min":0.20300000000000001,"15min":0.192},"max":{"1min":5.5490000000000004,"5min":25.128,"15min":25.128},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.47599999999999998,"15min":0.48399999999999999},"min":{"1min":0.22900000000000001,"5min":0.214,"15min":0.20100000000000001},"max":{"1min":5.2709999999999999,"5min":5.2709999999999999,"15min":6.7800000000000002},"last":0.40899999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739426,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6191104,"kb_used_data":4472180,"kb_used_omap":417,"kb_used_meta":1718494,"kb_avail":88180736,"statfs":{"total":96636764160,"available":90297073664,"internally_reserved":0,"allocated":4579512320,"data_stored":5441915491,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":427766,"internal_metadata":1759738122},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,4,6,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:10:07 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.59299999999999997,"15min":0.47999999999999998},"min":{"1min":0.19400000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":3.9689999999999999,"5min":6.1360000000000001,"15min":6.1360000000000001},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.621,"15min":0.50900000000000001},"min":{"1min":0.223,"5min":0.19600000000000001,"15min":0.191},"max":{"1min":4.4180000000000001,"5min":6.5910000000000002,"15min":7.4880000000000004},"last":0.32200000000000001}]},{"osd":2,"last update":"Tue Oct 17 02:10:18 2023","interfaces":[{"interface":"back","average":{"1min":0.29199999999999998,"5min":0.73999999999999999,"15min":0.58199999999999996},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.47199999999999998,"5min":12.382,"15min":12.382},"last":0.28699999999999998},{"interface":"front","average":{"1min":1.0149999999999999,"5min":0.78200000000000003,"15min":0.64000000000000001},"min":{"1min":0.248,"5min":0.23100000000000001,"15min":0.20499999999999999},"max":{"1min":5.4029999999999996,"5min":12.561,"15min":12.561},"last":0.35399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4578656256,"data_stored":5441253376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4437905408,"data_stored":5275738131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9016090624,"data_stored":10716520467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:10:20.401 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:10:20.401 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:10:40.403 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:10:41.219 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:10:41.220 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:10:41.254 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2768,"stamp":"2023-10-17T02:10:39.612852+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28692511576,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2237589,"num_read_kb":8748421,"num_write":2209407,"num_write_kb":8837279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87977,"ondisk_log_size":87977,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23244740,"kb_used_data":17671788,"kb_used_omap":1614,"kb_used_meta":5571313,"kb_avail":259870780,"statfs":{"total":289910292480,"available":266107678720,"internally_reserved":0,"allocated":18095910912,"data_stored":21495655804,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1653485,"internal_metadata":5705024787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,11,8,9,12,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19111936,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4637,"num_read_kb":18496,"num_write":4664,"num_write_kb":18656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002034"},"pg_stats":[{"pgid":"2.7","version":"114'14614","reported_seq":20525,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:33.053832+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:10:33.053832+0000","last_peered":"2023-10-17T02:10:33.053832+0000","last_clean":"2023-10-17T02:10:33.053832+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:10:33.053832+0000","last_undegraded":"2023-10-17T02:10:33.053832+0000","last_fullsized":"2023-10-17T02:10:33.053832+0000","mapping_epoch":13,"log_start":"83'4600","ondisk_log_start":"83'4600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"114'11007","reported_seq":16616,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:33.040252+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:10:33.040252+0000","last_peered":"2023-10-17T02:10:33.040252+0000","last_clean":"2023-10-17T02:10:33.040252+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:10:33.040252+0000","last_undegraded":"2023-10-17T02:10:33.040252+0000","last_fullsized":"2023-10-17T02:10:33.040252+0000","mapping_epoch":13,"log_start":"50'1000","ondisk_log_start":"50'1000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"114'7378","reported_seq":10428,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:33.026808+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:10:33.026808+0000","last_peered":"2023-10-17T02:10:33.026808+0000","last_clean":"2023-10-17T02:10:33.026808+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:10:33.026808+0000","last_undegraded":"2023-10-17T02:10:33.026808+0000","last_fullsized":"2023-10-17T02:10:33.026808+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":7378,"log_dups_size":0,"ondisk_log_size":7378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"114'11042","reported_seq":16061,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:33.049124+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:10:33.049124+0000","last_peered":"2023-10-17T02:10:33.049124+0000","last_clean":"2023-10-17T02:10:33.049124+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:10:33.049124+0000","last_undegraded":"2023-10-17T02:10:33.049124+0000","last_fullsized":"2023-10-17T02:10:33.049124+0000","mapping_epoch":13,"log_start":"50'1000","ondisk_log_start":"50'1000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"114'563143","reported_seq":1123046,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:39.102411+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:10:39.102411+0000","last_peered":"2023-10-17T02:10:39.102411+0000","last_clean":"2023-10-17T02:10:39.102411+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:10:39.102411+0000","last_undegraded":"2023-10-17T02:10:39.102411+0000","last_fullsized":"2023-10-17T02:10:39.102411+0000","mapping_epoch":16,"log_start":"113'553110","ondisk_log_start":"113'553110","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7275085824,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":560582,"num_read_kb":2209392,"num_write":560386,"num_write_kb":2241544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":274,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:09:57.471837+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:09:57.471837+0000","last_peered":"2023-10-17T02:09:57.471837+0000","last_clean":"2023-10-17T02:09:57.471837+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:09:57.471837+0000","last_undegraded":"2023-10-17T02:09:57.471837+0000","last_fullsized":"2023-10-17T02:09:57.471837+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"114'541224","reported_seq":1080249,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:34.830709+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:10:34.830709+0000","last_peered":"2023-10-17T02:10:34.830709+0000","last_clean":"2023-10-17T02:10:34.830709+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:10:34.830709+0000","last_undegraded":"2023-10-17T02:10:34.830709+0000","last_fullsized":"2023-10-17T02:10:34.830709+0000","mapping_epoch":16,"log_start":"112'531219","ondisk_log_start":"112'531219","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":6993395712,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":539702,"num_read_kb":2129096,"num_write":538574,"num_write_kb":2154296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":3013,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:32.135219+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:10:32.135219+0000","last_peered":"2023-10-17T02:10:32.135219+0000","last_clean":"2023-10-17T02:10:32.135219+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:10:32.135219+0000","last_undegraded":"2023-10-17T02:10:32.135219+0000","last_fullsized":"2023-10-17T02:10:32.135219+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"114'564025","reported_seq":1125905,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:39.071505+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:10:39.071505+0000","last_peered":"2023-10-17T02:10:39.071505+0000","last_clean":"2023-10-17T02:10:39.071505+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:10:39.071505+0000","last_undegraded":"2023-10-17T02:10:39.071505+0000","last_fullsized":"2023-10-17T02:10:39.071505+0000","mapping_epoch":16,"log_start":"113'553925","ondisk_log_start":"113'553925","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7291461632,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":562595,"num_read_kb":2218276,"num_write":561271,"num_write_kb":2245084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4513,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:32.068007+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:10:32.068007+0000","last_peered":"2023-10-17T02:10:32.068007+0000","last_clean":"2023-10-17T02:10:32.068007+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:10:32.068007+0000","last_undegraded":"2023-10-17T02:10:32.068007+0000","last_fullsized":"2023-10-17T02:10:32.068007+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"114'550678","reported_seq":1100839,"reported_epoch":114,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:10:34.850994+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:10:34.850994+0000","last_peered":"2023-10-17T02:10:34.850994+0000","last_clean":"2023-10-17T02:10:34.850994+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:10:34.850994+0000","last_undegraded":"2023-10-17T02:10:34.850994+0000","last_fullsized":"2023-10-17T02:10:34.850994+0000","mapping_epoch":16,"log_start":"112'540618","ondisk_log_start":"112'540618","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1557,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 549s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7131975699,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":550748,"num_read_kb":2171004,"num_write":548546,"num_write_kb":2194178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"114'11011","reported_seq":15394,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:33.071598+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:10:33.071598+0000","last_peered":"2023-10-17T02:10:33.071598+0000","last_clean":"2023-10-17T02:10:33.071598+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:10:33.071598+0000","last_undegraded":"2023-10-17T02:10:33.071598+0000","last_fullsized":"2023-10-17T02:10:33.071598+0000","mapping_epoch":13,"log_start":"50'1000","ondisk_log_start":"50'1000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":294,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-17T02:08:47.455867+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:08:47.455867+0000","last_peered":"2023-10-17T02:08:47.455867+0000","last_clean":"2023-10-17T02:08:47.455867+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:08:47.455867+0000","last_undegraded":"2023-10-17T02:08:47.455867+0000","last_fullsized":"2023-10-17T02:08:47.455867+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28691918867,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2213627,"num_read_kb":8727768,"num_write":2208777,"num_write_kb":8835102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18094088192,"data_stored":21494947878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40198,"ondisk_log_size":40198,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34264,"internal_metadata":0},"log_size":47680,"ondisk_log_size":47680,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739433,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10887164,"kb_used_data":8834932,"kb_used_omap":832,"kb_used_meta":2051391,"kb_avail":83484676,"statfs":{"total":96636764160,"available":85488308224,"internally_reserved":0,"allocated":9046970368,"data_stored":10746754429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":852977,"internal_metadata":2100624399},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:10:33 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.41799999999999998,"15min":0.44700000000000001},"min":{"1min":0.184,"5min":0.158,"15min":0.158},"max":{"1min":2.161,"5min":2.8479999999999999,"15min":12.852},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.38200000000000001,"15min":0.50900000000000001},"min":{"1min":0.219,"5min":0.20499999999999999,"15min":0.192},"max":{"1min":0.55100000000000005,"5min":2.8090000000000002,"15min":12.784000000000001},"last":0.48899999999999999}]},{"osd":1,"last update":"Tue Oct 17 02:10:06 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.47999999999999998,"15min":0.48499999999999999},"min":{"1min":0.224,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":4.0730000000000004,"5min":4.6980000000000004,"15min":9.5429999999999993},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.52300000000000002,"15min":0.60699999999999998},"min":{"1min":0.23999999999999999,"5min":0.23799999999999999,"15min":0.23100000000000001},"max":{"1min":5.1779999999999999,"5min":5.1779999999999999,"15min":12.824999999999999},"last":0.42699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739429,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6075836,"kb_used_data":4349492,"kb_used_omap":364,"kb_used_meta":1725971,"kb_avail":88296004,"statfs":{"total":96636764160,"available":90415108096,"internally_reserved":0,"allocated":4453879808,"data_stored":5291437468,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":372742,"internal_metadata":1767395322},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,7,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:10:29 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.77900000000000003,"15min":0.57799999999999996},"min":{"1min":0.182,"5min":0.16900000000000001,"15min":0.16300000000000001},"max":{"1min":4.1520000000000001,"5min":11.891999999999999,"15min":11.891999999999999},"last":0.31},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.82499999999999996,"15min":0.626},"min":{"1min":0.247,"5min":0.224,"15min":0.22},"max":{"1min":4.2329999999999997,"5min":11.926,"15min":11.926},"last":0.34499999999999997}]},{"osd":2,"last update":"Tue Oct 17 02:10:29 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.90100000000000002,"15min":0.67300000000000004},"min":{"1min":0.26600000000000001,"5min":0.20300000000000001,"15min":0.19500000000000001},"max":{"1min":0.90600000000000003,"5min":25.128,"15min":25.128},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.47099999999999997,"15min":0.48099999999999998},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.20100000000000001},"max":{"1min":0.52400000000000002,"5min":5.2709999999999999,"15min":6.7800000000000002},"last":0.33000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739430,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6281740,"kb_used_data":4487364,"kb_used_omap":417,"kb_used_meta":1793950,"kb_avail":88090100,"statfs":{"total":96636764160,"available":90204262400,"internally_reserved":0,"allocated":4595060736,"data_stored":5457463907,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":427766,"internal_metadata":1837005066},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,8,1,0,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:10:07 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.59299999999999997,"15min":0.47999999999999998},"min":{"1min":0.19400000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":3.9689999999999999,"5min":6.1360000000000001,"15min":6.1360000000000001},"last":1.768},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.621,"15min":0.50900000000000001},"min":{"1min":0.223,"5min":0.19600000000000001,"15min":0.191},"max":{"1min":4.4180000000000001,"5min":6.5910000000000002,"15min":7.4880000000000004},"last":0.22700000000000001}]},{"osd":2,"last update":"Tue Oct 17 02:10:18 2023","interfaces":[{"interface":"back","average":{"1min":0.29199999999999998,"5min":0.73999999999999999,"15min":0.58199999999999996},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.47199999999999998,"5min":12.382,"15min":12.382},"last":0.29099999999999998},{"interface":"front","average":{"1min":1.0149999999999999,"5min":0.78200000000000003,"15min":0.64000000000000001},"min":{"1min":0.248,"5min":0.23100000000000001,"15min":0.20499999999999999},"max":{"1min":5.4029999999999996,"5min":12.561,"15min":12.561},"last":0.26600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7225,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4594315264,"data_stored":5456912384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4452843520,"data_stored":5290676243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9046929408,"data_stored":10747359251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:10:41.256 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:10:41.257 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:11:01.258 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:11:01.787 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:11:01.788 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:11:01.814 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2779,"stamp":"2023-10-17T02:11:01.618415+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28727749464,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2246025,"num_read_kb":8782101,"num_write":2217989,"num_write_kb":8871607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88210,"ondisk_log_size":88210,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23019888,"kb_used_data":17735832,"kb_used_omap":1604,"kb_used_meta":5282427,"kb_avail":260095632,"statfs":{"total":289910292480,"available":266337927168,"internally_reserved":0,"allocated":18161491968,"data_stored":21561236860,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1643450,"internal_metadata":5409205318},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,4,5,30,10,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19562496,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4658,"num_read_kb":18596,"num_write":4776,"num_write_kb":19104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003868"},"pg_stats":[{"pgid":"2.7","version":"114'14674","reported_seq":20605,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:59.377075+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:10:59.377075+0000","last_peered":"2023-10-17T02:10:59.377075+0000","last_clean":"2023-10-17T02:10:59.377075+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:10:59.377075+0000","last_undegraded":"2023-10-17T02:10:59.377075+0000","last_fullsized":"2023-10-17T02:10:59.377075+0000","mapping_epoch":13,"log_start":"83'4600","ondisk_log_start":"83'4600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"114'11052","reported_seq":16676,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:59.370118+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:10:59.370118+0000","last_peered":"2023-10-17T02:10:59.370118+0000","last_clean":"2023-10-17T02:10:59.370118+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:10:59.370118+0000","last_undegraded":"2023-10-17T02:10:59.370118+0000","last_fullsized":"2023-10-17T02:10:59.370118+0000","mapping_epoch":13,"log_start":"50'1000","ondisk_log_start":"50'1000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"114'7408","reported_seq":10468,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:59.354912+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:10:59.354912+0000","last_peered":"2023-10-17T02:10:59.354912+0000","last_clean":"2023-10-17T02:10:59.354912+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:10:59.354912+0000","last_undegraded":"2023-10-17T02:10:59.354912+0000","last_fullsized":"2023-10-17T02:10:59.354912+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":7408,"log_dups_size":0,"ondisk_log_size":7408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"114'11087","reported_seq":16121,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:59.370062+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:10:59.370062+0000","last_peered":"2023-10-17T02:10:59.370062+0000","last_clean":"2023-10-17T02:10:59.370062+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:10:59.370062+0000","last_undegraded":"2023-10-17T02:10:59.370062+0000","last_fullsized":"2023-10-17T02:10:59.370062+0000","mapping_epoch":13,"log_start":"50'1000","ondisk_log_start":"50'1000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"114'565176","reported_seq":1126974,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:59.230100+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:10:59.230100+0000","last_peered":"2023-10-17T02:10:59.230100+0000","last_clean":"2023-10-17T02:10:59.230100+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:10:59.230100+0000","last_undegraded":"2023-10-17T02:10:59.230100+0000","last_fullsized":"2023-10-17T02:10:59.230100+0000","mapping_epoch":16,"log_start":"113'555110","ondisk_log_start":"113'555110","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7283384320,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":562486,"num_read_kb":2216988,"num_write":562419,"num_write_kb":2249676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":274,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:09:57.471837+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:09:57.471837+0000","last_peered":"2023-10-17T02:09:57.471837+0000","last_clean":"2023-10-17T02:09:57.471837+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:09:57.471837+0000","last_undegraded":"2023-10-17T02:09:57.471837+0000","last_fullsized":"2023-10-17T02:09:57.471837+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"114'543484","reported_seq":1084816,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:59.998029+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:10:59.998029+0000","last_peered":"2023-10-17T02:10:59.998029+0000","last_clean":"2023-10-17T02:10:59.998029+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:10:59.998029+0000","last_undegraded":"2023-10-17T02:10:59.998029+0000","last_fullsized":"2023-10-17T02:10:59.998029+0000","mapping_epoch":16,"log_start":"113'533419","ondisk_log_start":"113'533419","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":7002644480,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":542008,"num_read_kb":2138308,"num_write":540834,"num_write_kb":2163336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":3021,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:52.547163+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:10:52.547163+0000","last_peered":"2023-10-17T02:10:52.547163+0000","last_clean":"2023-10-17T02:10:52.547163+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:10:52.547163+0000","last_undegraded":"2023-10-17T02:10:52.547163+0000","last_fullsized":"2023-10-17T02:10:52.547163+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"114'565927","reported_seq":1129679,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:59.230469+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:10:59.230469+0000","last_peered":"2023-10-17T02:10:59.230469+0000","last_clean":"2023-10-17T02:10:59.230469+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:10:59.230469+0000","last_undegraded":"2023-10-17T02:10:59.230469+0000","last_fullsized":"2023-10-17T02:10:59.230469+0000","mapping_epoch":16,"log_start":"113'555925","ondisk_log_start":"113'555925","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7299350528,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":564470,"num_read_kb":2225756,"num_write":563173,"num_write_kb":2252692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4526,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:52.569751+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:10:52.569751+0000","last_peered":"2023-10-17T02:10:52.569751+0000","last_clean":"2023-10-17T02:10:52.569751+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:10:52.569751+0000","last_undegraded":"2023-10-17T02:10:52.569751+0000","last_fullsized":"2023-10-17T02:10:52.569751+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"114'553091","reported_seq":1105617,"reported_epoch":114,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:10:59.988132+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:10:59.988132+0000","last_peered":"2023-10-17T02:10:59.988132+0000","last_clean":"2023-10-17T02:10:59.988132+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:10:59.988132+0000","last_undegraded":"2023-10-17T02:10:59.988132+0000","last_fullsized":"2023-10-17T02:10:59.988132+0000","mapping_epoch":16,"log_start":"113'543018","ondisk_log_start":"113'543018","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1605,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 574s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7141777427,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":553099,"num_read_kb":2180396,"num_write":550933,"num_write_kb":2203726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"114'11056","reported_seq":15454,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:59.397949+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:10:59.397949+0000","last_peered":"2023-10-17T02:10:59.397949+0000","last_clean":"2023-10-17T02:10:59.397949+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:10:59.397949+0000","last_undegraded":"2023-10-17T02:10:59.397949+0000","last_fullsized":"2023-10-17T02:10:59.397949+0000","mapping_epoch":13,"log_start":"50'1000","ondisk_log_start":"50'1000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":295,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:09:57.469526+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:09:57.469526+0000","last_peered":"2023-10-17T02:09:57.469526+0000","last_clean":"2023-10-17T02:09:57.469526+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:09:57.469526+0000","last_undegraded":"2023-10-17T02:09:57.469526+0000","last_fullsized":"2023-10-17T02:09:57.469526+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28727156755,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2222063,"num_read_kb":8761448,"num_write":2217359,"num_write_kb":8869430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18160349184,"data_stored":21561208870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40206,"ondisk_log_size":40206,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34223,"internal_metadata":0},"log_size":47905,"ondisk_log_size":47905,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739437,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10515908,"kb_used_data":8864956,"kb_used_omap":823,"kb_used_meta":1650120,"kb_avail":83855932,"statfs":{"total":96636764160,"available":85868474368,"internally_reserved":0,"allocated":9077714944,"data_stored":10777499005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842942,"internal_metadata":1689723714},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,2,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:10:33 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.41799999999999998,"15min":0.44700000000000001},"min":{"1min":0.184,"5min":0.158,"15min":0.158},"max":{"1min":2.161,"5min":2.8479999999999999,"15min":12.852},"last":2.3900000000000001},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.38200000000000001,"15min":0.50900000000000001},"min":{"1min":0.219,"5min":0.20499999999999999,"15min":0.192},"max":{"1min":0.55100000000000005,"5min":2.8090000000000002,"15min":12.784000000000001},"last":0.254}]},{"osd":1,"last update":"Tue Oct 17 02:10:06 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.47999999999999998,"15min":0.48499999999999999},"min":{"1min":0.224,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":4.0730000000000004,"5min":4.6980000000000004,"15min":9.5429999999999993},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.52300000000000002,"15min":0.60699999999999998},"min":{"1min":0.23999999999999999,"5min":0.23799999999999999,"15min":0.23100000000000001},"max":{"1min":5.1779999999999999,"5min":5.1779999999999999,"15min":12.824999999999999},"last":0.35299999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739434,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6152484,"kb_used_data":4367900,"kb_used_omap":364,"kb_used_meta":1784211,"kb_avail":88219356,"statfs":{"total":96636764160,"available":90336620544,"internally_reserved":0,"allocated":4472729600,"data_stored":5310287260,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":372742,"internal_metadata":1827033082},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,25,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:10:29 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.77900000000000003,"15min":0.57799999999999996},"min":{"1min":0.182,"5min":0.16900000000000001,"15min":0.16300000000000001},"max":{"1min":4.1520000000000001,"5min":11.891999999999999,"15min":11.891999999999999},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.82499999999999996,"15min":0.626},"min":{"1min":0.247,"5min":0.224,"15min":0.22},"max":{"1min":4.2329999999999997,"5min":11.926,"15min":11.926},"last":1.0229999999999999}]},{"osd":2,"last update":"Tue Oct 17 02:10:29 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.90100000000000002,"15min":0.67300000000000004},"min":{"1min":0.26600000000000001,"5min":0.20300000000000001,"15min":0.19500000000000001},"max":{"1min":0.90600000000000003,"5min":25.128,"15min":25.128},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.47099999999999997,"15min":0.48099999999999998},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.20100000000000001},"max":{"1min":0.52400000000000002,"5min":5.2709999999999999,"15min":6.7800000000000002},"last":0.437}]}]},{"osd":0,"up_from":8,"seq":34359739434,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6351496,"kb_used_data":4502976,"kb_used_omap":417,"kb_used_meta":1848094,"kb_avail":88020344,"statfs":{"total":96636764160,"available":90132832256,"internally_reserved":0,"allocated":4611047424,"data_stored":5473450595,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":427766,"internal_metadata":1892448522},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,1,3,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:10:07 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.59299999999999997,"15min":0.47999999999999998},"min":{"1min":0.19400000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":3.9689999999999999,"5min":6.1360000000000001,"15min":6.1360000000000001},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.621,"15min":0.50900000000000001},"min":{"1min":0.223,"5min":0.19600000000000001,"15min":0.191},"max":{"1min":4.4180000000000001,"5min":6.5910000000000002,"15min":7.4880000000000004},"last":0.23100000000000001}]},{"osd":2,"last update":"Tue Oct 17 02:10:18 2023","interfaces":[{"interface":"back","average":{"1min":0.29199999999999998,"5min":0.73999999999999999,"15min":0.58199999999999996},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.47199999999999998,"5min":12.382,"15min":12.382},"last":0.39600000000000002},{"interface":"front","average":{"1min":1.0149999999999999,"5min":0.78200000000000003,"15min":0.64000000000000001},"min":{"1min":0.248,"5min":0.23100000000000001,"15min":0.20499999999999999},"max":{"1min":5.4029999999999996,"5min":12.561,"15min":12.561},"last":0.28299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7184,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4610433024,"data_stored":5473030144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4471881728,"data_stored":5309714451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9078034432,"data_stored":10778464275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:11:01.816 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:11:01.816 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:11:21.818 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:11:22.334 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:11:22.334 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:11:22.369 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2790,"stamp":"2023-10-17T02:11:21.621797+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28758379352,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2253230,"num_read_kb":8810833,"num_write":2225474,"num_write_kb":8901547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88213,"ondisk_log_size":88213,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23294720,"kb_used_data":17795232,"kb_used_omap":1604,"kb_used_meta":5497851,"kb_avail":259820800,"statfs":{"total":289910292480,"available":266056499200,"internally_reserved":0,"allocated":18222317568,"data_stored":21622053142,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1643450,"internal_metadata":5629799494},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,9,8,13,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15175680,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3612,"num_read_kb":14412,"num_write":3711,"num_write_kb":14844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001678"},"pg_stats":[{"pgid":"2.7","version":"114'14726","reported_seq":20674,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:18.143982+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:11:18.143982+0000","last_peered":"2023-10-17T02:11:18.143982+0000","last_clean":"2023-10-17T02:11:18.143982+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:11:18.143982+0000","last_undegraded":"2023-10-17T02:11:18.143982+0000","last_fullsized":"2023-10-17T02:11:18.143982+0000","mapping_epoch":13,"log_start":"85'4700","ondisk_log_start":"85'4700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"114'11091","reported_seq":16729,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:18.144013+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:11:18.144013+0000","last_peered":"2023-10-17T02:11:18.144013+0000","last_clean":"2023-10-17T02:11:18.144013+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:11:18.144013+0000","last_undegraded":"2023-10-17T02:11:18.144013+0000","last_fullsized":"2023-10-17T02:11:18.144013+0000","mapping_epoch":13,"log_start":"50'1000","ondisk_log_start":"50'1000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"114'7434","reported_seq":10503,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:18.143303+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:11:18.143303+0000","last_peered":"2023-10-17T02:11:18.143303+0000","last_clean":"2023-10-17T02:11:18.143303+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:11:18.143303+0000","last_undegraded":"2023-10-17T02:11:18.143303+0000","last_fullsized":"2023-10-17T02:11:18.143303+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":7434,"log_dups_size":0,"ondisk_log_size":7434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"114'11126","reported_seq":16173,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:18.143747+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:11:18.143747+0000","last_peered":"2023-10-17T02:11:18.143747+0000","last_clean":"2023-10-17T02:11:18.143747+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:11:18.143747+0000","last_undegraded":"2023-10-17T02:11:18.143747+0000","last_fullsized":"2023-10-17T02:11:18.143747+0000","mapping_epoch":13,"log_start":"52'1100","ondisk_log_start":"52'1100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"115'567073","reported_seq":1130679,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:19.361195+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:11:19.361195+0000","last_peered":"2023-10-17T02:11:19.361195+0000","last_clean":"2023-10-17T02:11:19.361195+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:11:19.361195+0000","last_undegraded":"2023-10-17T02:11:19.361195+0000","last_fullsized":"2023-10-17T02:11:19.361195+0000","mapping_epoch":16,"log_start":"113'557010","ondisk_log_start":"113'557010","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7291117568,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":564285,"num_read_kb":2224168,"num_write":564316,"num_write_kb":2257264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":275,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:13.942640+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:10:13.942640+0000","last_peered":"2023-10-17T02:10:13.942640+0000","last_clean":"2023-10-17T02:10:13.942640+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:10:13.942640+0000","last_undegraded":"2023-10-17T02:10:13.942640+0000","last_fullsized":"2023-10-17T02:10:13.942640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"115'545331","reported_seq":1088386,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:20.115964+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:11:20.115964+0000","last_peered":"2023-10-17T02:11:20.115964+0000","last_clean":"2023-10-17T02:11:20.115964+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:11:20.115964+0000","last_undegraded":"2023-10-17T02:11:20.115964+0000","last_fullsized":"2023-10-17T02:11:20.115964+0000","mapping_epoch":16,"log_start":"113'535319","ondisk_log_start":"113'535319","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":7010177024,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":543732,"num_read_kb":2145184,"num_write":542681,"num_write_kb":2170724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":3029,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:11:13.027155+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:11:13.027155+0000","last_peered":"2023-10-17T02:11:13.027155+0000","last_clean":"2023-10-17T02:11:13.027155+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:11:13.027155+0000","last_undegraded":"2023-10-17T02:11:13.027155+0000","last_fullsized":"2023-10-17T02:11:13.027155+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"115'567805","reported_seq":1133383,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:19.351315+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:11:19.351315+0000","last_peered":"2023-10-17T02:11:19.351315+0000","last_clean":"2023-10-17T02:11:19.351315+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:11:19.351315+0000","last_undegraded":"2023-10-17T02:11:19.351315+0000","last_fullsized":"2023-10-17T02:11:19.351315+0000","mapping_epoch":16,"log_start":"113'557725","ondisk_log_start":"113'557725","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7307112448,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":566291,"num_read_kb":2233004,"num_write":565051,"num_write_kb":2260204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4538,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:11:13.027564+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:11:13.027564+0000","last_peered":"2023-10-17T02:11:13.027564+0000","last_clean":"2023-10-17T02:11:13.027564+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:11:13.027564+0000","last_undegraded":"2023-10-17T02:11:13.027564+0000","last_fullsized":"2023-10-17T02:11:13.027564+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"115'554977","reported_seq":1109371,"reported_epoch":115,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:11:20.117980+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:11:20.117980+0000","last_peered":"2023-10-17T02:11:20.117980+0000","last_clean":"2023-10-17T02:11:20.117980+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:11:20.117980+0000","last_undegraded":"2023-10-17T02:11:20.117980+0000","last_fullsized":"2023-10-17T02:11:20.117980+0000","mapping_epoch":16,"log_start":"113'544918","ondisk_log_start":"113'544918","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1643,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 594s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7149379603,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":554960,"num_read_kb":2187824,"num_write":552796,"num_write_kb":2211178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"114'11095","reported_seq":15506,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:18.143593+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:11:18.143593+0000","last_peered":"2023-10-17T02:11:18.143593+0000","last_clean":"2023-10-17T02:11:18.143593+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:11:18.143593+0000","last_undegraded":"2023-10-17T02:11:18.143593+0000","last_fullsized":"2023-10-17T02:11:18.143593+0000","mapping_epoch":13,"log_start":"50'1000","ondisk_log_start":"50'1000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":296,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:44.900440+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:10:44.900440+0000","last_peered":"2023-10-17T02:10:44.900440+0000","last_clean":"2023-10-17T02:10:44.900440+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:10:44.900440+0000","last_undegraded":"2023-10-17T02:10:44.900440+0000","last_fullsized":"2023-10-17T02:10:44.900440+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28757786643,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2229268,"num_read_kb":8790180,"num_write":2224844,"num_write_kb":8899370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18221453312,"data_stored":21622312998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40214,"ondisk_log_size":40214,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34223,"internal_metadata":0},"log_size":47900,"ondisk_log_size":47900,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739441,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10656948,"kb_used_data":8894696,"kb_used_omap":823,"kb_used_meta":1761416,"kb_avail":83714892,"statfs":{"total":96636764160,"available":85724049408,"internally_reserved":0,"allocated":9108168704,"data_stored":10807952765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842942,"internal_metadata":1803690818},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:10:33 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.41799999999999998,"15min":0.44700000000000001},"min":{"1min":0.184,"5min":0.158,"15min":0.158},"max":{"1min":2.161,"5min":2.8479999999999999,"15min":12.852},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.38200000000000001,"15min":0.50900000000000001},"min":{"1min":0.219,"5min":0.20499999999999999,"15min":0.192},"max":{"1min":0.55100000000000005,"5min":2.8090000000000002,"15min":12.784000000000001},"last":0.48899999999999999}]},{"osd":1,"last update":"Tue Oct 17 02:11:08 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.47499999999999998,"15min":0.497},"min":{"1min":0.223,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.9449999999999998,"5min":4.6980000000000004,"15min":9.5429999999999993},"last":1.498},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.51200000000000001,"15min":0.59999999999999998},"min":{"1min":0.255,"5min":0.23799999999999999,"15min":0.23100000000000001},"max":{"1min":2.1850000000000001,"5min":5.1779999999999999,"15min":12.824999999999999},"last":1.6100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739438,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6217124,"kb_used_data":4382556,"kb_used_omap":364,"kb_used_meta":1834195,"kb_avail":88154716,"statfs":{"total":96636764160,"available":90270429184,"internally_reserved":0,"allocated":4487737344,"data_stored":5325290345,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":372742,"internal_metadata":1878216698},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,2,6,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:10:29 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.77900000000000003,"15min":0.57799999999999996},"min":{"1min":0.182,"5min":0.16900000000000001,"15min":0.16300000000000001},"max":{"1min":4.1520000000000001,"5min":11.891999999999999,"15min":11.891999999999999},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.82499999999999996,"15min":0.626},"min":{"1min":0.247,"5min":0.224,"15min":0.22},"max":{"1min":4.2329999999999997,"5min":11.926,"15min":11.926},"last":0.36799999999999999}]},{"osd":2,"last update":"Tue Oct 17 02:10:29 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.90100000000000002,"15min":0.67300000000000004},"min":{"1min":0.26600000000000001,"5min":0.20300000000000001,"15min":0.19500000000000001},"max":{"1min":0.90600000000000003,"5min":25.128,"15min":25.128},"last":1.641},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.47099999999999997,"15min":0.48099999999999998},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.20100000000000001},"max":{"1min":0.52400000000000002,"5min":5.2709999999999999,"15min":6.7800000000000002},"last":1.4930000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739438,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6420648,"kb_used_data":4517980,"kb_used_omap":417,"kb_used_meta":1902238,"kb_avail":87951192,"statfs":{"total":96636764160,"available":90062020608,"internally_reserved":0,"allocated":4626411520,"data_stored":5488810032,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":427766,"internal_metadata":1947891978},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:11:10 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.53700000000000003,"15min":0.48299999999999998},"min":{"1min":0.19,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.768,"5min":5.4370000000000003,"15min":6.1360000000000001},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.29799999999999999,"5min":0.52400000000000002,"15min":0.50900000000000001},"min":{"1min":0.219,"5min":0.19600000000000001,"15min":0.191},"max":{"1min":0.44400000000000001,"5min":5.5119999999999996,"15min":7.4880000000000004},"last":0.40600000000000003}]},{"osd":2,"last update":"Tue Oct 17 02:10:18 2023","interfaces":[{"interface":"back","average":{"1min":0.29199999999999998,"5min":0.73999999999999999,"15min":0.58199999999999996},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.47199999999999998,"5min":12.382,"15min":12.382},"last":0.36199999999999999},{"interface":"front","average":{"1min":1.0149999999999999,"5min":0.78200000000000003,"15min":0.64000000000000001},"min":{"1min":0.248,"5min":0.23100000000000001,"15min":0.20499999999999999},"max":{"1min":5.4029999999999996,"5min":12.561,"15min":12.561},"last":0.433}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7184,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4625895424,"data_stored":5488492544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4487073792,"data_stored":5324906515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9108484096,"data_stored":10808913939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:11:22.371 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:11:22.371 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-17T02:11:22.948 INFO:teuthology.orchestra.run.smithi169.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-17T02:11:22.980 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:11:30.214 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:30.207+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T02:11:30.223 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:30.215+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T02:11:30.233 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:30.227+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T02:11:30.233 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:30.227+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T02:11:30.233 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:30.227+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T02:11:30.234 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:30.227+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T02:11:30.234 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:30.227+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T02:11:30.234 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:30.227+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T02:11:30.235 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:30.227+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T02:11:30.235 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:30.227+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T02:11:30.235 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:30.227+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T02:11:30.235 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:30.227+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T02:11:30.236 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:30.227+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T02:11:30.236 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:30.227+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T02:11:39.252 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:39.239+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T02:11:39.262 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:39.251+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T02:11:39.269 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:39.263+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T02:11:39.272 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:39.267+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T02:11:39.284 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:39.279+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T02:11:39.284 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:39.279+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T02:11:39.299 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:39.291+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T02:11:39.312 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:39.303+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T02:11:39.312 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:39.303+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T02:11:39.312 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:39.303+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T02:11:39.313 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:39.303+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T02:11:39.313 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:39.303+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T02:11:39.313 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:39.303+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T02:11:42.981 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:11:43.471 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:11:43.472 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:11:43.505 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2800,"stamp":"2023-10-17T02:11:41.627452+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28788525912,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2260563,"num_read_kb":8840057,"num_write":2232885,"num_write_kb":8931191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88296,"ondisk_log_size":88296,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23610636,"kb_used_data":17855796,"kb_used_omap":1604,"kb_used_meta":5753211,"kb_avail":259504884,"statfs":{"total":289910292480,"available":265733001216,"internally_reserved":0,"allocated":18284335104,"data_stored":21684066019,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1643450,"internal_metadata":5891288134},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,1,9,13,30,25,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18800640,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4551,"num_read_kb":18128,"num_write":4617,"num_write_kb":18468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004294"},"pg_stats":[{"pgid":"2.7","version":"115'14789","reported_seq":20754,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:39.307109+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:11:39.307109+0000","last_peered":"2023-10-17T02:11:39.307109+0000","last_clean":"2023-10-17T02:11:39.307109+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:11:39.307109+0000","last_undegraded":"2023-10-17T02:11:39.307109+0000","last_fullsized":"2023-10-17T02:11:39.307109+0000","mapping_epoch":13,"log_start":"85'4700","ondisk_log_start":"85'4700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"115'11140","reported_seq":16790,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:39.304068+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:11:39.304068+0000","last_peered":"2023-10-17T02:11:39.304068+0000","last_clean":"2023-10-17T02:11:39.304068+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:11:39.304068+0000","last_undegraded":"2023-10-17T02:11:39.304068+0000","last_fullsized":"2023-10-17T02:11:39.304068+0000","mapping_epoch":13,"log_start":"52'1100","ondisk_log_start":"52'1100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"115'7468","reported_seq":10545,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:39.290258+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:11:39.290258+0000","last_peered":"2023-10-17T02:11:39.290258+0000","last_clean":"2023-10-17T02:11:39.290258+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:11:39.290258+0000","last_undegraded":"2023-10-17T02:11:39.290258+0000","last_fullsized":"2023-10-17T02:11:39.290258+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":7468,"log_dups_size":0,"ondisk_log_size":7468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"115'11174","reported_seq":16233,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:39.311097+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:11:39.311097+0000","last_peered":"2023-10-17T02:11:39.311097+0000","last_clean":"2023-10-17T02:11:39.311097+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:11:39.311097+0000","last_undegraded":"2023-10-17T02:11:39.311097+0000","last_fullsized":"2023-10-17T02:11:39.311097+0000","mapping_epoch":13,"log_start":"52'1100","ondisk_log_start":"52'1100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"115'568953","reported_seq":1134390,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:39.461096+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:11:39.461096+0000","last_peered":"2023-10-17T02:11:39.461096+0000","last_clean":"2023-10-17T02:11:39.461096+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:11:39.461096+0000","last_undegraded":"2023-10-17T02:11:39.461096+0000","last_fullsized":"2023-10-17T02:11:39.461096+0000","mapping_epoch":16,"log_start":"113'558910","ondisk_log_start":"113'558910","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7298723840,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":566115,"num_read_kb":2231460,"num_write":566196,"num_write_kb":2264784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":275,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:13.942640+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:10:13.942640+0000","last_peered":"2023-10-17T02:10:13.942640+0000","last_clean":"2023-10-17T02:10:13.942640+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:10:13.942640+0000","last_undegraded":"2023-10-17T02:10:13.942640+0000","last_fullsized":"2023-10-17T02:10:13.942640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"115'547109","reported_seq":1091924,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:40.253401+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:11:40.253401+0000","last_peered":"2023-10-17T02:11:40.253401+0000","last_clean":"2023-10-17T02:11:40.253401+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:11:40.253401+0000","last_undegraded":"2023-10-17T02:11:40.253401+0000","last_fullsized":"2023-10-17T02:11:40.253401+0000","mapping_epoch":16,"log_start":"113'537019","ondisk_log_start":"113'537019","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":7017406464,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":545491,"num_read_kb":2152196,"num_write":544459,"num_write_kb":2177836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":3038,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:33.532698+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:11:33.532698+0000","last_peered":"2023-10-17T02:11:33.532698+0000","last_clean":"2023-10-17T02:11:33.532698+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:11:33.532698+0000","last_undegraded":"2023-10-17T02:11:33.532698+0000","last_fullsized":"2023-10-17T02:11:33.532698+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"115'569687","reported_seq":1137142,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:39.456013+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:11:39.456013+0000","last_peered":"2023-10-17T02:11:39.456013+0000","last_clean":"2023-10-17T02:11:39.456013+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:11:39.456013+0000","last_undegraded":"2023-10-17T02:11:39.456013+0000","last_fullsized":"2023-10-17T02:11:39.456013+0000","mapping_epoch":16,"log_start":"113'559625","ondisk_log_start":"113'559625","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7314759680,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":568166,"num_read_kb":2240476,"num_write":566933,"num_write_kb":2267732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4551,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:33.566060+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:11:33.566060+0000","last_peered":"2023-10-17T02:11:33.566060+0000","last_clean":"2023-10-17T02:11:33.566060+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:11:33.566060+0000","last_undegraded":"2023-10-17T02:11:33.566060+0000","last_fullsized":"2023-10-17T02:11:33.566060+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"115'556876","reported_seq":1113150,"reported_epoch":115,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:11:40.273989+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:11:40.273989+0000","last_peered":"2023-10-17T02:11:40.273989+0000","last_clean":"2023-10-17T02:11:40.273989+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:11:40.273989+0000","last_undegraded":"2023-10-17T02:11:40.273989+0000","last_fullsized":"2023-10-17T02:11:40.273989+0000","mapping_epoch":16,"log_start":"113'546818","ondisk_log_start":"113'546818","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1691,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 614s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7157043219,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":556829,"num_read_kb":2195272,"num_write":554667,"num_write_kb":2218662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"115'11145","reported_seq":15568,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:39.328495+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:11:39.328495+0000","last_peered":"2023-10-17T02:11:39.328495+0000","last_clean":"2023-10-17T02:11:39.328495+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:11:39.328495+0000","last_undegraded":"2023-10-17T02:11:39.328495+0000","last_fullsized":"2023-10-17T02:11:39.328495+0000","mapping_epoch":13,"log_start":"52'1100","ondisk_log_start":"52'1100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":296,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:44.900440+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:10:44.900440+0000","last_peered":"2023-10-17T02:10:44.900440+0000","last_clean":"2023-10-17T02:10:44.900440+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:10:44.900440+0000","last_undegraded":"2023-10-17T02:10:44.900440+0000","last_fullsized":"2023-10-17T02:10:44.900440+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28787933203,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2236601,"num_read_kb":8819404,"num_write":2232255,"num_write_kb":8929014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18282340352,"data_stored":21683200038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40253,"ondisk_log_size":40253,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34223,"internal_metadata":0},"log_size":47944,"ondisk_log_size":47944,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739445,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10819388,"kb_used_data":8924916,"kb_used_omap":823,"kb_used_meta":1893640,"kb_avail":83552452,"statfs":{"total":96636764160,"available":85557710848,"internally_reserved":0,"allocated":9139113984,"data_stored":10838893386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842942,"internal_metadata":1939088194},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:10:33 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.41799999999999998,"15min":0.44700000000000001},"min":{"1min":0.184,"5min":0.158,"15min":0.158},"max":{"1min":2.161,"5min":2.8479999999999999,"15min":12.852},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.38200000000000001,"15min":0.50900000000000001},"min":{"1min":0.219,"5min":0.20499999999999999,"15min":0.192},"max":{"1min":0.55100000000000005,"5min":2.8090000000000002,"15min":12.784000000000001},"last":0.27100000000000002}]},{"osd":1,"last update":"Tue Oct 17 02:11:08 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.47499999999999998,"15min":0.497},"min":{"1min":0.223,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.9449999999999998,"5min":4.6980000000000004,"15min":9.5429999999999993},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.51200000000000001,"15min":0.59999999999999998},"min":{"1min":0.255,"5min":0.23799999999999999,"15min":0.23100000000000001},"max":{"1min":2.1850000000000001,"5min":5.1779999999999999,"15min":12.824999999999999},"last":2.819}]}]},{"osd":1,"up_from":8,"seq":34359739442,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6311764,"kb_used_data":4397708,"kb_used_omap":364,"kb_used_meta":1913683,"kb_avail":88060076,"statfs":{"total":96636764160,"available":90173517824,"internally_reserved":0,"allocated":4503252992,"data_stored":5340805993,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":372742,"internal_metadata":1959612410},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,9,26,20,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:11:31 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.82099999999999995,"15min":0.57599999999999996},"min":{"1min":0.19800000000000001,"5min":0.16900000000000001,"15min":0.16300000000000001},"max":{"1min":4.0709999999999997,"5min":11.891999999999999,"15min":11.891999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.86399999999999999,"15min":0.624},"min":{"1min":0.25900000000000001,"5min":0.224,"15min":0.22},"max":{"1min":4.1210000000000004,"5min":11.926,"15min":11.926},"last":0.61199999999999999}]},{"osd":2,"last update":"Tue Oct 17 02:11:31 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.20599999999999999,"5min":0.20300000000000001,"15min":0.19500000000000001},"max":{"1min":6.3680000000000003,"5min":8.0570000000000004,"15min":25.128},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.47099999999999997,"15min":0.48399999999999999},"min":{"1min":0.219,"5min":0.214,"15min":0.20100000000000001},"max":{"1min":1.4930000000000001,"5min":5.2709999999999999,"15min":6.7800000000000002},"last":0.26100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739442,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6479484,"kb_used_data":4533172,"kb_used_omap":417,"kb_used_meta":1945886,"kb_avail":87892356,"statfs":{"total":96636764160,"available":90001772544,"internally_reserved":0,"allocated":4641968128,"data_stored":5504366640,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":427766,"internal_metadata":1992587530},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,1,3,3,2,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:11:10 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.53700000000000003,"15min":0.48299999999999998},"min":{"1min":0.19,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.768,"5min":5.4370000000000003,"15min":6.1360000000000001},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.29799999999999999,"5min":0.52400000000000002,"15min":0.50900000000000001},"min":{"1min":0.219,"5min":0.19600000000000001,"15min":0.191},"max":{"1min":0.44400000000000001,"5min":5.5119999999999996,"15min":7.4880000000000004},"last":0.23200000000000001}]},{"osd":2,"last update":"Tue Oct 17 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.55400000000000005,"15min":0.58799999999999997},"min":{"1min":0.21299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":1.8129999999999999,"5min":12.382,"15min":12.382},"last":0.25},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.73099999999999998,"15min":0.65200000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":3.8780000000000001,"5min":12.561,"15min":12.561},"last":0.26700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7184,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4641312768,"data_stored":5503909888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4502028288,"data_stored":5339861011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9138999296,"data_stored":10839429139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:11:43.507 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:11:43.507 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:11:47.676 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:47.671+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T02:11:47.676 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:47.671+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T02:11:47.677 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:47.671+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbda750 2023-10-17T02:11:47.682 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:47.675+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T02:11:47.682 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:47.675+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T02:11:47.682 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:47.675+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T02:11:47.683 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:47.675+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T02:11:47.683 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:47.675+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T02:11:47.683 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:47.675+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T02:11:47.683 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:47.675+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T02:11:47.684 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:47.675+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T02:11:47.684 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:47.675+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T02:11:47.684 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:47.675+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T02:11:47.685 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:47.675+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T02:11:47.685 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:47.675+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T02:11:47.981 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:47.971+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T02:11:47.995 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:47.987+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T02:11:47.995 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:47.987+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T02:11:47.996 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:47.987+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T02:11:47.999 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:11:47.991+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T02:12:03.508 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:12:04.093 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:12:04.094 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:12:04.139 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2811,"stamp":"2023-10-17T02:12:03.631569+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28819069784,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2268141,"num_read_kb":8870253,"num_write":2240317,"num_write_kb":8960919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88222,"ondisk_log_size":88222,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23400184,"kb_used_data":17922784,"kb_used_omap":1642,"kb_used_meta":5475733,"kb_avail":259715336,"statfs":{"total":289910292480,"available":265948504064,"internally_reserved":0,"allocated":18352930816,"data_stored":21752661731,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1681414,"internal_metadata":5607151610},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,3,6,4,9,18,15],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15376384,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3855,"num_read_kb":15364,"num_write":3752,"num_write_kb":15008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002206"},"pg_stats":[{"pgid":"2.7","version":"115'14858","reported_seq":20839,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:59.986427+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:11:59.986427+0000","last_peered":"2023-10-17T02:11:59.986427+0000","last_clean":"2023-10-17T02:11:59.986427+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:11:59.986427+0000","last_undegraded":"2023-10-17T02:11:59.986427+0000","last_fullsized":"2023-10-17T02:11:59.986427+0000","mapping_epoch":13,"log_start":"85'4800","ondisk_log_start":"85'4800","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"115'11192","reported_seq":16854,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:59.981518+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:11:59.981518+0000","last_peered":"2023-10-17T02:11:59.981518+0000","last_clean":"2023-10-17T02:11:59.981518+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:11:59.981518+0000","last_undegraded":"2023-10-17T02:11:59.981518+0000","last_fullsized":"2023-10-17T02:11:59.981518+0000","mapping_epoch":13,"log_start":"52'1100","ondisk_log_start":"52'1100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"115'7503","reported_seq":10588,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:59.963981+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:11:59.963981+0000","last_peered":"2023-10-17T02:11:59.963981+0000","last_clean":"2023-10-17T02:11:59.963981+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:11:59.963981+0000","last_undegraded":"2023-10-17T02:11:59.963981+0000","last_fullsized":"2023-10-17T02:11:59.963981+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":7503,"log_dups_size":0,"ondisk_log_size":7503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"115'11226","reported_seq":16297,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:59.974373+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:11:59.974373+0000","last_peered":"2023-10-17T02:11:59.974373+0000","last_clean":"2023-10-17T02:11:59.974373+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:11:59.974373+0000","last_undegraded":"2023-10-17T02:11:59.974373+0000","last_fullsized":"2023-10-17T02:11:59.974373+0000","mapping_epoch":13,"log_start":"52'1200","ondisk_log_start":"52'1200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"115'570835","reported_seq":1138223,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:59.602013+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:11:59.602013+0000","last_peered":"2023-10-17T02:11:59.602013+0000","last_clean":"2023-10-17T02:11:59.602013+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:11:59.602013+0000","last_undegraded":"2023-10-17T02:11:59.602013+0000","last_fullsized":"2023-10-17T02:11:59.602013+0000","mapping_epoch":16,"log_start":"114'560810","ondisk_log_start":"114'560810","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7306412032,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":568067,"num_read_kb":2239236,"num_write":568078,"num_write_kb":2272312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":275,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:13.942640+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:10:13.942640+0000","last_peered":"2023-10-17T02:10:13.942640+0000","last_clean":"2023-10-17T02:10:13.942640+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:10:13.942640+0000","last_undegraded":"2023-10-17T02:10:13.942640+0000","last_fullsized":"2023-10-17T02:10:13.942640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"115'548926","reported_seq":1095552,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:12:00.359571+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:12:00.359571+0000","last_peered":"2023-10-17T02:12:00.359571+0000","last_clean":"2023-10-17T02:12:00.359571+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:12:00.359571+0000","last_undegraded":"2023-10-17T02:12:00.359571+0000","last_fullsized":"2023-10-17T02:12:00.359571+0000","mapping_epoch":16,"log_start":"114'538919","ondisk_log_start":"114'538919","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":7024832512,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":547301,"num_read_kb":2159420,"num_write":546276,"num_write_kb":2185104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":3048,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:59.104875+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:11:59.104875+0000","last_peered":"2023-10-17T02:11:59.104875+0000","last_clean":"2023-10-17T02:11:59.104875+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:11:59.104875+0000","last_undegraded":"2023-10-17T02:11:59.104875+0000","last_fullsized":"2023-10-17T02:11:59.104875+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"115'571565","reported_seq":1140978,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:59.600883+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:11:59.600883+0000","last_peered":"2023-10-17T02:11:59.600883+0000","last_clean":"2023-10-17T02:11:59.600883+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:11:59.600883+0000","last_undegraded":"2023-10-17T02:11:59.600883+0000","last_fullsized":"2023-10-17T02:11:59.600883+0000","mapping_epoch":16,"log_start":"114'561525","ondisk_log_start":"114'561525","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7322550272,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":570124,"num_read_kb":2248280,"num_write":568811,"num_write_kb":2275244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4566,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:59.130427+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:11:59.130427+0000","last_peered":"2023-10-17T02:11:59.130427+0000","last_clean":"2023-10-17T02:11:59.130427+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:11:59.130427+0000","last_undegraded":"2023-10-17T02:11:59.130427+0000","last_fullsized":"2023-10-17T02:11:59.130427+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"115'558765","reported_seq":1116909,"reported_epoch":115,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:12:00.402494+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:12:00.402494+0000","last_peered":"2023-10-17T02:12:00.402494+0000","last_clean":"2023-10-17T02:12:00.402494+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:12:00.402494+0000","last_undegraded":"2023-10-17T02:12:00.402494+0000","last_fullsized":"2023-10-17T02:12:00.402494+0000","mapping_epoch":16,"log_start":"114'548718","ondisk_log_start":"114'548718","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1757,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 634s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7164682259,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":558687,"num_read_kb":2202664,"num_write":556522,"num_write_kb":2226082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"115'11197","reported_seq":15632,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:59.994666+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:11:59.994666+0000","last_peered":"2023-10-17T02:11:59.994666+0000","last_clean":"2023-10-17T02:11:59.994666+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:11:59.994666+0000","last_undegraded":"2023-10-17T02:11:59.994666+0000","last_fullsized":"2023-10-17T02:11:59.994666+0000","mapping_epoch":13,"log_start":"52'1100","ondisk_log_start":"52'1100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":296,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:44.900440+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:10:44.900440+0000","last_peered":"2023-10-17T02:10:44.900440+0000","last_clean":"2023-10-17T02:10:44.900440+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:10:44.900440+0000","last_undegraded":"2023-10-17T02:10:44.900440+0000","last_fullsized":"2023-10-17T02:10:44.900440+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28818477075,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2244179,"num_read_kb":8849600,"num_write":2239687,"num_write_kb":8958742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18349817856,"data_stored":21750677542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40119,"ondisk_log_size":40119,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34223,"internal_metadata":0},"log_size":48004,"ondisk_log_size":48004,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739450,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11011796,"kb_used_data":8961804,"kb_used_omap":823,"kb_used_meta":2049160,"kb_avail":83360044,"statfs":{"total":96636764160,"available":85360685056,"internally_reserved":0,"allocated":9176887296,"data_stored":10876666698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842942,"internal_metadata":2098340674},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,2,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:11:38 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.434,"15min":0.45200000000000001},"min":{"1min":0.186,"5min":0.158,"15min":0.158},"max":{"1min":2.3900000000000001,"5min":2.3900000000000001,"15min":12.852},"last":0.187},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.374,"15min":0.48399999999999999},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.192},"max":{"1min":1.706,"5min":1.738,"15min":12.784000000000001},"last":0.371}]},{"osd":1,"last update":"Tue Oct 17 02:11:08 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.47499999999999998,"15min":0.497},"min":{"1min":0.223,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.9449999999999998,"5min":4.6980000000000004,"15min":9.5429999999999993},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.51200000000000001,"15min":0.59999999999999998},"min":{"1min":0.255,"5min":0.23799999999999999,"15min":0.23100000000000001},"max":{"1min":2.1850000000000001,"5min":5.1779999999999999,"15min":12.824999999999999},"last":0.316}]}]},{"osd":1,"up_from":8,"seq":34359739446,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6362456,"kb_used_data":4412304,"kb_used_omap":364,"kb_used_meta":1949779,"kb_avail":88009384,"statfs":{"total":96636764160,"available":90121609216,"internally_reserved":0,"allocated":4518199296,"data_stored":5355752297,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":372742,"internal_metadata":1996574714},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,0,3,11,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:11:31 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.82099999999999995,"15min":0.57599999999999996},"min":{"1min":0.19800000000000001,"5min":0.16900000000000001,"15min":0.16300000000000001},"max":{"1min":4.0709999999999997,"5min":11.891999999999999,"15min":11.891999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.86399999999999999,"15min":0.624},"min":{"1min":0.25900000000000001,"5min":0.224,"15min":0.22},"max":{"1min":4.1210000000000004,"5min":11.926,"15min":11.926},"last":0.56100000000000005}]},{"osd":2,"last update":"Tue Oct 17 02:11:31 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.20599999999999999,"5min":0.20300000000000001,"15min":0.19500000000000001},"max":{"1min":6.3680000000000003,"5min":8.0570000000000004,"15min":25.128},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.47099999999999997,"15min":0.48399999999999999},"min":{"1min":0.219,"5min":0.214,"15min":0.20100000000000001},"max":{"1min":1.4930000000000001,"5min":5.2709999999999999,"15min":6.7800000000000002},"last":0.71299999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739446,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6025932,"kb_used_data":4548676,"kb_used_omap":454,"kb_used_meta":1476793,"kb_avail":88345908,"statfs":{"total":96636764160,"available":90466209792,"internally_reserved":0,"allocated":4657844224,"data_stored":5520242736,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":465730,"internal_metadata":1512236222},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,2,1,4,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:11:10 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.53700000000000003,"15min":0.48299999999999998},"min":{"1min":0.19,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.768,"5min":5.4370000000000003,"15min":6.1360000000000001},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.29799999999999999,"5min":0.52400000000000002,"15min":0.50900000000000001},"min":{"1min":0.219,"5min":0.19600000000000001,"15min":0.191},"max":{"1min":0.44400000000000001,"5min":5.5119999999999996,"15min":7.4880000000000004},"last":0.44700000000000001}]},{"osd":2,"last update":"Tue Oct 17 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.55400000000000005,"15min":0.58799999999999997},"min":{"1min":0.21299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":1.8129999999999999,"5min":12.382,"15min":12.382},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.73099999999999998,"15min":0.65200000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":3.8780000000000001,"5min":12.561,"15min":12.561},"last":0.28899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7184,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4656697344,"data_stored":5519294464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4517056512,"data_stored":5354889235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9176064000,"data_stored":10876493843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:12:04.140 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:12:04.141 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:12:24.142 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:12:24.683 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:12:24.685 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:12:24.715 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2821,"stamp":"2023-10-17T02:12:23.638410+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28850019160,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2275690,"num_read_kb":8900373,"num_write":2247797,"num_write_kb":8990839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88230,"ondisk_log_size":88230,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22657648,"kb_used_data":17982612,"kb_used_omap":1671,"kb_used_meta":4673336,"kb_avail":260457872,"statfs":{"total":289910292480,"available":266708860928,"internally_reserved":0,"allocated":18414194688,"data_stored":21813925603,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1711939,"internal_metadata":4785496253},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,2,5,11,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15425536,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3737,"num_read_kb":14920,"num_write":3721,"num_write_kb":14884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005512"},"pg_stats":[{"pgid":"2.7","version":"115'14910","reported_seq":20907,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:12:19.589019+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:12:19.589019+0000","last_peered":"2023-10-17T02:12:19.589019+0000","last_clean":"2023-10-17T02:12:19.589019+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:12:19.589019+0000","last_undegraded":"2023-10-17T02:12:19.589019+0000","last_fullsized":"2023-10-17T02:12:19.589019+0000","mapping_epoch":13,"log_start":"87'4900","ondisk_log_start":"87'4900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"115'11231","reported_seq":16905,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:12:19.589310+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:12:19.589310+0000","last_peered":"2023-10-17T02:12:19.589310+0000","last_clean":"2023-10-17T02:12:19.589310+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:12:19.589310+0000","last_undegraded":"2023-10-17T02:12:19.589310+0000","last_fullsized":"2023-10-17T02:12:19.589310+0000","mapping_epoch":13,"log_start":"53'1200","ondisk_log_start":"53'1200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"115'7529","reported_seq":10623,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:12:19.588742+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:12:19.588742+0000","last_peered":"2023-10-17T02:12:19.588742+0000","last_clean":"2023-10-17T02:12:19.588742+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:12:19.588742+0000","last_undegraded":"2023-10-17T02:12:19.588742+0000","last_fullsized":"2023-10-17T02:12:19.588742+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":7529,"log_dups_size":0,"ondisk_log_size":7529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"115'11265","reported_seq":16349,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:12:19.588991+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:12:19.588991+0000","last_peered":"2023-10-17T02:12:19.588991+0000","last_clean":"2023-10-17T02:12:19.588991+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:12:19.588991+0000","last_undegraded":"2023-10-17T02:12:19.588991+0000","last_fullsized":"2023-10-17T02:12:19.588991+0000","mapping_epoch":13,"log_start":"52'1200","ondisk_log_start":"52'1200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"115'572686","reported_seq":1141986,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:12:19.771164+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:12:19.771164+0000","last_peered":"2023-10-17T02:12:19.771164+0000","last_clean":"2023-10-17T02:12:19.771164+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:12:19.771164+0000","last_undegraded":"2023-10-17T02:12:19.771164+0000","last_fullsized":"2023-10-17T02:12:19.771164+0000","mapping_epoch":16,"log_start":"114'562610","ondisk_log_start":"114'562610","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7314083840,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":569984,"num_read_kb":2246884,"num_write":569929,"num_write_kb":2279716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":276,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:17.493344+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:11:17.493344+0000","last_peered":"2023-10-17T02:11:17.493344+0000","last_clean":"2023-10-17T02:11:17.493344+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:11:17.493344+0000","last_undegraded":"2023-10-17T02:11:17.493344+0000","last_fullsized":"2023-10-17T02:11:17.493344+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"115'550791","reported_seq":1099230,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:12:20.474344+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:12:20.474344+0000","last_peered":"2023-10-17T02:12:20.474344+0000","last_clean":"2023-10-17T02:12:20.474344+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:12:20.474344+0000","last_undegraded":"2023-10-17T02:12:20.474344+0000","last_fullsized":"2023-10-17T02:12:20.474344+0000","mapping_epoch":16,"log_start":"114'540719","ondisk_log_start":"114'540719","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":7032541184,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":549113,"num_read_kb":2166656,"num_write":548141,"num_write_kb":2192564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":3057,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:12:19.588794+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:12:19.588794+0000","last_peered":"2023-10-17T02:12:19.588794+0000","last_clean":"2023-10-17T02:12:19.588794+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:12:19.588794+0000","last_undegraded":"2023-10-17T02:12:19.588794+0000","last_fullsized":"2023-10-17T02:12:19.588794+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"115'573466","reported_seq":1144827,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:12:19.757733+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:12:19.757733+0000","last_peered":"2023-10-17T02:12:19.757733+0000","last_clean":"2023-10-17T02:12:19.757733+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:12:19.757733+0000","last_undegraded":"2023-10-17T02:12:19.757733+0000","last_fullsized":"2023-10-17T02:12:19.757733+0000","mapping_epoch":16,"log_start":"114'563425","ondisk_log_start":"114'563425","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7330439168,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":572074,"num_read_kb":2256064,"num_write":570712,"num_write_kb":2282848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4578,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:12:19.584075+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:12:19.584075+0000","last_peered":"2023-10-17T02:12:19.584075+0000","last_clean":"2023-10-17T02:12:19.584075+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:12:19.584075+0000","last_undegraded":"2023-10-17T02:12:19.584075+0000","last_fullsized":"2023-10-17T02:12:19.584075+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"115'560661","reported_seq":1120682,"reported_epoch":115,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:12:20.475394+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:12:20.475394+0000","last_peered":"2023-10-17T02:12:20.475394+0000","last_clean":"2023-10-17T02:12:20.475394+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:12:20.475394+0000","last_undegraded":"2023-10-17T02:12:20.475394+0000","last_fullsized":"2023-10-17T02:12:20.475394+0000","mapping_epoch":16,"log_start":"114'550618","ondisk_log_start":"114'550618","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1817,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 655s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7172362259,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":560557,"num_read_kb":2210116,"num_write":558385,"num_write_kb":2233534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"115'11236","reported_seq":15684,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:12:19.651767+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:12:19.651767+0000","last_peered":"2023-10-17T02:12:19.651767+0000","last_clean":"2023-10-17T02:12:19.651767+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:12:19.651767+0000","last_undegraded":"2023-10-17T02:12:19.651767+0000","last_fullsized":"2023-10-17T02:12:19.651767+0000","mapping_epoch":13,"log_start":"54'1200","ondisk_log_start":"54'1200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":296,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:44.900440+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:10:44.900440+0000","last_peered":"2023-10-17T02:10:44.900440+0000","last_clean":"2023-10-17T02:10:44.900440+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:10:44.900440+0000","last_undegraded":"2023-10-17T02:10:44.900440+0000","last_fullsized":"2023-10-17T02:10:44.900440+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28849426451,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2251728,"num_read_kb":8879720,"num_write":2247167,"num_write_kb":8988662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18411151360,"data_stored":21812011046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40232,"ondisk_log_size":40232,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34223,"internal_metadata":0},"log_size":47899,"ondisk_log_size":47899,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739454,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10167272,"kb_used_data":8991840,"kb_used_omap":852,"kb_used_meta":1174571,"kb_avail":84204568,"statfs":{"total":96636764160,"available":86225477632,"internally_reserved":0,"allocated":9207644160,"data_stored":10907423562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":873467,"internal_metadata":1202760709},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:11:38 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.434,"15min":0.45200000000000001},"min":{"1min":0.186,"5min":0.158,"15min":0.158},"max":{"1min":2.3900000000000001,"5min":2.3900000000000001,"15min":12.852},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.374,"15min":0.48399999999999999},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.192},"max":{"1min":1.706,"5min":1.738,"15min":12.784000000000001},"last":0.35699999999999998}]},{"osd":1,"last update":"Tue Oct 17 02:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.496,"15min":0.47199999999999998},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.498,"5min":4.6980000000000004,"15min":6.202},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.56499999999999995,"15min":0.58599999999999997},"min":{"1min":0.26900000000000002,"5min":0.23999999999999999,"15min":0.23200000000000001},"max":{"1min":2.819,"5min":5.1779999999999999,"15min":12.824999999999999},"last":0.312}]}]},{"osd":1,"up_from":8,"seq":34359739450,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6413496,"kb_used_data":4427244,"kb_used_omap":364,"kb_used_meta":1985875,"kb_avail":87958344,"statfs":{"total":96636764160,"available":90069344256,"internally_reserved":0,"allocated":4533497856,"data_stored":5371050857,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":372742,"internal_metadata":2033537018},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,5,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:11:31 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.82099999999999995,"15min":0.57599999999999996},"min":{"1min":0.19800000000000001,"5min":0.16900000000000001,"15min":0.16300000000000001},"max":{"1min":4.0709999999999997,"5min":11.891999999999999,"15min":11.891999999999999},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.86399999999999999,"15min":0.624},"min":{"1min":0.25900000000000001,"5min":0.224,"15min":0.22},"max":{"1min":4.1210000000000004,"5min":11.926,"15min":11.926},"last":0.38300000000000001}]},{"osd":2,"last update":"Tue Oct 17 02:11:31 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.20599999999999999,"5min":0.20300000000000001,"15min":0.19500000000000001},"max":{"1min":6.3680000000000003,"5min":8.0570000000000004,"15min":25.128},"last":0.218},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.47099999999999997,"15min":0.48399999999999999},"min":{"1min":0.219,"5min":0.214,"15min":0.20100000000000001},"max":{"1min":1.4930000000000001,"5min":5.2709999999999999,"15min":6.7800000000000002},"last":0.24199999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739450,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6076880,"kb_used_data":4563528,"kb_used_omap":454,"kb_used_meta":1512889,"kb_avail":88294960,"statfs":{"total":96636764160,"available":90414039040,"internally_reserved":0,"allocated":4673052672,"data_stored":5535451184,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":465730,"internal_metadata":1549198526},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:12:13 2023","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":0.64500000000000002,"15min":0.54100000000000004},"min":{"1min":0.19500000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":13.834,"5min":13.834,"15min":13.834},"last":0.23999999999999999},{"interface":"front","average":{"1min":1.0920000000000001,"5min":0.64900000000000002,"15min":0.55800000000000005},"min":{"1min":0.23200000000000001,"5min":0.19600000000000001,"15min":0.191},"max":{"1min":9.0730000000000004,"5min":9.0730000000000004,"15min":9.0730000000000004},"last":0.218}]},{"osd":2,"last update":"Tue Oct 17 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.55400000000000005,"15min":0.58799999999999997},"min":{"1min":0.21299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":1.8129999999999999,"5min":12.382,"15min":12.382},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.73099999999999998,"15min":0.65200000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":3.8780000000000001,"5min":12.561,"15min":12.561},"last":0.38200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7184,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4672065536,"data_stored":5534662656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4532338688,"data_stored":5370171411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9206747136,"data_stored":10907176979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:12:24.717 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:12:24.717 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:12:44.719 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:12:45.195 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:12:45.197 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:12:45.233 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2831,"stamp":"2023-10-17T02:12:43.644521+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28881005400,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2283306,"num_read_kb":8930729,"num_write":2255297,"num_write_kb":9020839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88360,"ondisk_log_size":88360,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22928752,"kb_used_data":18042196,"kb_used_omap":1671,"kb_used_meta":4884856,"kb_avail":260186768,"statfs":{"total":289910292480,"available":266431250432,"internally_reserved":0,"allocated":18475208704,"data_stored":21874939619,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1711939,"internal_metadata":5002092733},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,4,11,12,15,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15101952,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3714,"num_read_kb":14820,"num_write":3678,"num_write_kb":14712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002204"},"pg_stats":[{"pgid":"2.7","version":"115'14938","reported_seq":20951,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:12:40.067828+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:12:40.067828+0000","last_peered":"2023-10-17T02:12:40.067828+0000","last_clean":"2023-10-17T02:12:40.067828+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:12:40.067828+0000","last_undegraded":"2023-10-17T02:12:40.067828+0000","last_fullsized":"2023-10-17T02:12:40.067828+0000","mapping_epoch":13,"log_start":"87'4900","ondisk_log_start":"87'4900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"115'11252","reported_seq":16938,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:12:40.067532+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:12:40.067532+0000","last_peered":"2023-10-17T02:12:40.067532+0000","last_clean":"2023-10-17T02:12:40.067532+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:12:40.067532+0000","last_undegraded":"2023-10-17T02:12:40.067532+0000","last_fullsized":"2023-10-17T02:12:40.067532+0000","mapping_epoch":13,"log_start":"53'1200","ondisk_log_start":"53'1200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"115'7543","reported_seq":10645,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:12:40.067133+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:12:40.067133+0000","last_peered":"2023-10-17T02:12:40.067133+0000","last_clean":"2023-10-17T02:12:40.067133+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:12:40.067133+0000","last_undegraded":"2023-10-17T02:12:40.067133+0000","last_fullsized":"2023-10-17T02:12:40.067133+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":7543,"log_dups_size":0,"ondisk_log_size":7543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"115'11286","reported_seq":16382,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:12:40.067262+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:12:40.067262+0000","last_peered":"2023-10-17T02:12:40.067262+0000","last_clean":"2023-10-17T02:12:40.067262+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:12:40.067262+0000","last_undegraded":"2023-10-17T02:12:40.067262+0000","last_fullsized":"2023-10-17T02:12:40.067262+0000","mapping_epoch":13,"log_start":"52'1200","ondisk_log_start":"52'1200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"115'574579","reported_seq":1145778,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:12:39.897952+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:12:39.897952+0000","last_peered":"2023-10-17T02:12:39.897952+0000","last_clean":"2023-10-17T02:12:39.897952+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:12:39.897952+0000","last_undegraded":"2023-10-17T02:12:39.897952+0000","last_fullsized":"2023-10-17T02:12:39.897952+0000","mapping_epoch":16,"log_start":"114'564510","ondisk_log_start":"114'564510","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7322009600,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":571880,"num_read_kb":2254436,"num_write":571822,"num_write_kb":2287288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":276,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:17.493344+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:11:17.493344+0000","last_peered":"2023-10-17T02:11:17.493344+0000","last_clean":"2023-10-17T02:11:17.493344+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:11:17.493344+0000","last_undegraded":"2023-10-17T02:11:17.493344+0000","last_fullsized":"2023-10-17T02:11:17.493344+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"115'552602","reported_seq":1102859,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:12:40.609301+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:12:40.609301+0000","last_peered":"2023-10-17T02:12:40.609301+0000","last_clean":"2023-10-17T02:12:40.609301+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:12:40.609301+0000","last_undegraded":"2023-10-17T02:12:40.609301+0000","last_fullsized":"2023-10-17T02:12:40.609301+0000","mapping_epoch":16,"log_start":"114'542519","ondisk_log_start":"114'542519","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":7039963136,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":550932,"num_read_kb":2173908,"num_write":549952,"num_write_kb":2199808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":3065,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:12:40.070160+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:12:40.070160+0000","last_peered":"2023-10-17T02:12:40.070160+0000","last_clean":"2023-10-17T02:12:40.070160+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:12:40.070160+0000","last_undegraded":"2023-10-17T02:12:40.070160+0000","last_fullsized":"2023-10-17T02:12:40.070160+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"115'575378","reported_seq":1148677,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:12:39.895407+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:12:39.895407+0000","last_peered":"2023-10-17T02:12:39.895407+0000","last_clean":"2023-10-17T02:12:39.895407+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:12:39.895407+0000","last_undegraded":"2023-10-17T02:12:39.895407+0000","last_fullsized":"2023-10-17T02:12:39.895407+0000","mapping_epoch":16,"log_start":"114'565325","ondisk_log_start":"114'565325","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7338336256,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":574010,"num_read_kb":2263780,"num_write":572624,"num_write_kb":2290496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4590,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:12:40.071893+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:12:40.071893+0000","last_peered":"2023-10-17T02:12:40.071893+0000","last_clean":"2023-10-17T02:12:40.071893+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:12:40.071893+0000","last_undegraded":"2023-10-17T02:12:40.071893+0000","last_fullsized":"2023-10-17T02:12:40.071893+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"115'562570","reported_seq":1124565,"reported_epoch":115,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:12:40.596820+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:12:40.596820+0000","last_peered":"2023-10-17T02:12:40.596820+0000","last_clean":"2023-10-17T02:12:40.596820+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:12:40.596820+0000","last_undegraded":"2023-10-17T02:12:40.596820+0000","last_fullsized":"2023-10-17T02:12:40.596820+0000","mapping_epoch":16,"log_start":"114'552518","ondisk_log_start":"114'552518","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1863,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 675s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7180103699,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":562522,"num_read_kb":2217952,"num_write":560269,"num_write_kb":2241070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"115'11257","reported_seq":15717,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:12:40.067238+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:12:40.067238+0000","last_peered":"2023-10-17T02:12:40.067238+0000","last_clean":"2023-10-17T02:12:40.067238+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:12:40.067238+0000","last_undegraded":"2023-10-17T02:12:40.067238+0000","last_fullsized":"2023-10-17T02:12:40.067238+0000","mapping_epoch":13,"log_start":"54'1200","ondisk_log_start":"54'1200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":296,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-10-17T02:10:44.900440+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:10:44.900440+0000","last_peered":"2023-10-17T02:10:44.900440+0000","last_clean":"2023-10-17T02:10:44.900440+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:10:44.900440+0000","last_undegraded":"2023-10-17T02:10:44.900440+0000","last_fullsized":"2023-10-17T02:10:44.900440+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28880412691,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2259344,"num_read_kb":8910076,"num_write":2254667,"num_write_kb":9018662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18472538112,"data_stored":21873397798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40257,"ondisk_log_size":40257,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34223,"internal_metadata":0},"log_size":48004,"ondisk_log_size":48004,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739458,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10283112,"kb_used_data":9021600,"kb_used_omap":852,"kb_used_meta":1260651,"kb_avail":84088728,"statfs":{"total":96636764160,"available":86106857472,"internally_reserved":0,"allocated":9238118400,"data_stored":10937897802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":873467,"internal_metadata":1290906629},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,1,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:12:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.436,"15min":0.45900000000000002},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":1.8919999999999999,"5min":2.3900000000000001,"15min":12.852},"last":0.25},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.38800000000000001,"15min":0.48199999999999998},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.192},"max":{"1min":1.9410000000000001,"5min":1.9410000000000001,"15min":12.784000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Tue Oct 17 02:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.496,"15min":0.47199999999999998},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.498,"5min":4.6980000000000004,"15min":6.202},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.56499999999999995,"15min":0.58599999999999997},"min":{"1min":0.26900000000000002,"5min":0.23999999999999999,"15min":0.23200000000000001},"max":{"1min":2.819,"5min":5.1779999999999999,"15min":12.824999999999999},"last":0.378}]}]},{"osd":1,"up_from":8,"seq":34359739454,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6503620,"kb_used_data":4441912,"kb_used_omap":364,"kb_used_meta":2061331,"kb_avail":87868220,"statfs":{"total":96636764160,"available":89977057280,"internally_reserved":0,"allocated":4548517888,"data_stored":5386070889,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":372742,"internal_metadata":2110803962},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,8,3,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:12:33 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.61699999999999999,"15min":0.58799999999999997},"min":{"1min":0.17999999999999999,"5min":0.17100000000000001,"15min":0.16300000000000001},"max":{"1min":2.4750000000000001,"5min":4.2480000000000002,"15min":11.891999999999999},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.66200000000000003,"15min":0.63500000000000001},"min":{"1min":0.24099999999999999,"5min":0.224,"15min":0.22},"max":{"1min":2.5230000000000001,"5min":4.2869999999999999,"15min":11.926},"last":0.34399999999999997}]},{"osd":2,"last update":"Tue Oct 17 02:12:33 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.622,"15min":0.66500000000000004},"min":{"1min":0.20899999999999999,"5min":0.20300000000000001,"15min":0.19500000000000001},"max":{"1min":0.69399999999999995,"5min":6.3680000000000003,"15min":25.128},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.441,"15min":0.45700000000000002},"min":{"1min":0.24199999999999999,"5min":0.214,"15min":0.20100000000000001},"max":{"1min":0.71299999999999997,"5min":5.2709999999999999,"15min":6.7800000000000002},"last":0.36199999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739454,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6142020,"kb_used_data":4578684,"kb_used_omap":454,"kb_used_meta":1562873,"kb_avail":88229820,"statfs":{"total":96636764160,"available":90347335680,"internally_reserved":0,"allocated":4688572416,"data_stored":5550970928,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":465730,"internal_metadata":1600382142},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,5,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:12:13 2023","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":0.64500000000000002,"15min":0.54100000000000004},"min":{"1min":0.19500000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":13.834,"5min":13.834,"15min":13.834},"last":0.26300000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":0.64900000000000002,"15min":0.55800000000000005},"min":{"1min":0.23200000000000001,"5min":0.19600000000000001,"15min":0.191},"max":{"1min":9.0730000000000004,"5min":9.0730000000000004,"15min":9.0730000000000004},"last":0.32100000000000001}]},{"osd":2,"last update":"Tue Oct 17 02:12:24 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.55300000000000005,"15min":0.58099999999999996},"min":{"1min":0.20599999999999999,"5min":0.187,"15min":0.187},"max":{"1min":0.65300000000000002,"5min":12.382,"15min":12.382},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.73799999999999999,"15min":0.64900000000000002},"min":{"1min":0.26700000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":2.085,"5min":12.561,"15min":12.561},"last":0.28499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7184,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4687650816,"data_stored":5550247936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4547465216,"data_stored":5385297939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9237422080,"data_stored":10937851923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:12:45.235 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:12:45.236 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:13:05.237 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:13:05.878 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:13:05.880 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:13:05.942 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2842,"stamp":"2023-10-17T02:13:05.648318+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28915333976,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2291631,"num_read_kb":8963949,"num_write":2263719,"num_write_kb":9054527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88342,"ondisk_log_size":88342,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22601332,"kb_used_data":18105756,"kb_used_omap":1731,"kb_used_meta":4493820,"kb_avail":260514188,"statfs":{"total":289910292480,"available":266766528512,"internally_reserved":0,"allocated":18540294144,"data_stored":21940025059,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1773277,"internal_metadata":4601671971},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,6,7,8,9,8,18],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19116032,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4663,"num_read_kb":18612,"num_write":4705,"num_write_kb":18820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001967"},"pg_stats":[{"pgid":"2.7","version":"115'14974","reported_seq":21003,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:13:00.644121+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:13:00.644121+0000","last_peered":"2023-10-17T02:13:00.644121+0000","last_clean":"2023-10-17T02:13:00.644121+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:13:00.644121+0000","last_undegraded":"2023-10-17T02:13:00.644121+0000","last_fullsized":"2023-10-17T02:13:00.644121+0000","mapping_epoch":13,"log_start":"87'4900","ondisk_log_start":"87'4900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"115'11279","reported_seq":16977,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:13:00.630321+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:13:00.630321+0000","last_peered":"2023-10-17T02:13:00.630321+0000","last_clean":"2023-10-17T02:13:00.630321+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:13:00.630321+0000","last_undegraded":"2023-10-17T02:13:00.630321+0000","last_fullsized":"2023-10-17T02:13:00.630321+0000","mapping_epoch":13,"log_start":"53'1200","ondisk_log_start":"53'1200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"115'7561","reported_seq":10671,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:13:00.604553+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:13:00.604553+0000","last_peered":"2023-10-17T02:13:00.604553+0000","last_clean":"2023-10-17T02:13:00.604553+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:13:00.604553+0000","last_undegraded":"2023-10-17T02:13:00.604553+0000","last_fullsized":"2023-10-17T02:13:00.604553+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":7561,"log_dups_size":0,"ondisk_log_size":7561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"115'11313","reported_seq":16421,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:13:00.620416+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:13:00.620416+0000","last_peered":"2023-10-17T02:13:00.620416+0000","last_clean":"2023-10-17T02:13:00.620416+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:13:00.620416+0000","last_undegraded":"2023-10-17T02:13:00.620416+0000","last_fullsized":"2023-10-17T02:13:00.620416+0000","mapping_epoch":13,"log_start":"54'1300","ondisk_log_start":"54'1300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"115'576926","reported_seq":1150562,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:13:05.002041+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:13:05.002041+0000","last_peered":"2023-10-17T02:13:05.002041+0000","last_clean":"2023-10-17T02:13:05.002041+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:13:05.002041+0000","last_undegraded":"2023-10-17T02:13:05.002041+0000","last_fullsized":"2023-10-17T02:13:05.002041+0000","mapping_epoch":16,"log_start":"115'566910","ondisk_log_start":"115'566910","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7331598336,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":574315,"num_read_kb":2264156,"num_write":574169,"num_write_kb":2296676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":276,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:17.493344+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:11:17.493344+0000","last_peered":"2023-10-17T02:11:17.493344+0000","last_clean":"2023-10-17T02:11:17.493344+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:11:17.493344+0000","last_undegraded":"2023-10-17T02:11:17.493344+0000","last_fullsized":"2023-10-17T02:11:17.493344+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"115'554416","reported_seq":1106453,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:13:00.653950+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:13:00.653950+0000","last_peered":"2023-10-17T02:13:00.653950+0000","last_clean":"2023-10-17T02:13:00.653950+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:13:00.653950+0000","last_undegraded":"2023-10-17T02:13:00.653950+0000","last_fullsized":"2023-10-17T02:13:00.653950+0000","mapping_epoch":16,"log_start":"114'544319","ondisk_log_start":"114'544319","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":7047340032,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":552711,"num_read_kb":2181008,"num_write":551766,"num_write_kb":2207064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":3073,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:13:00.543260+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:13:00.543260+0000","last_peered":"2023-10-17T02:13:00.543260+0000","last_clean":"2023-10-17T02:13:00.543260+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:13:00.543260+0000","last_undegraded":"2023-10-17T02:13:00.543260+0000","last_fullsized":"2023-10-17T02:13:00.543260+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"115'577743","reported_seq":1153340,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:13:05.022462+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:13:05.022462+0000","last_peered":"2023-10-17T02:13:05.022462+0000","last_clean":"2023-10-17T02:13:05.022462+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:13:05.022462+0000","last_undegraded":"2023-10-17T02:13:05.022462+0000","last_fullsized":"2023-10-17T02:13:05.022462+0000","mapping_epoch":16,"log_start":"115'567725","ondisk_log_start":"115'567725","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7347986432,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":576307,"num_read_kb":2272944,"num_write":574989,"num_write_kb":2299956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4603,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:13:00.543456+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:13:00.543456+0000","last_peered":"2023-10-17T02:13:00.543456+0000","last_clean":"2023-10-17T02:13:00.543456+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:13:00.543456+0000","last_undegraded":"2023-10-17T02:13:00.543456+0000","last_fullsized":"2023-10-17T02:13:00.543456+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"115'564491","reported_seq":1128306,"reported_epoch":115,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:13:00.686560+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:13:00.686560+0000","last_peered":"2023-10-17T02:13:00.686560+0000","last_clean":"2023-10-17T02:13:00.686560+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:13:00.686560+0000","last_undegraded":"2023-10-17T02:13:00.686560+0000","last_fullsized":"2023-10-17T02:13:00.686560+0000","mapping_epoch":16,"log_start":"114'554418","ondisk_log_start":"114'554418","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1909,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 695s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7187816467,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":564336,"num_read_kb":2225188,"num_write":562165,"num_write_kb":2248654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"115'11284","reported_seq":15754,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:13:00.656129+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:13:00.656129+0000","last_peered":"2023-10-17T02:13:00.656129+0000","last_clean":"2023-10-17T02:13:00.656129+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:13:00.656129+0000","last_undegraded":"2023-10-17T02:13:00.656129+0000","last_fullsized":"2023-10-17T02:13:00.656129+0000","mapping_epoch":13,"log_start":"54'1200","ondisk_log_start":"54'1200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":297,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:17.499716+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:11:17.499716+0000","last_peered":"2023-10-17T02:11:17.499716+0000","last_clean":"2023-10-17T02:11:17.499716+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:11:17.499716+0000","last_undegraded":"2023-10-17T02:11:17.499716+0000","last_fullsized":"2023-10-17T02:11:17.499716+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28914741267,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2267669,"num_read_kb":8943296,"num_write":2263089,"num_write_kb":9052350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18537779200,"data_stored":21938638886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40204,"ondisk_log_size":40204,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34223,"internal_metadata":0},"log_size":48039,"ondisk_log_size":48039,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739462,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10441344,"kb_used_data":9051640,"kb_used_omap":852,"kb_used_meta":1388843,"kb_avail":83930496,"statfs":{"total":96636764160,"available":85944827904,"internally_reserved":0,"allocated":9268879360,"data_stored":10968658762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":873467,"internal_metadata":1422175237},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:12:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.436,"15min":0.45900000000000002},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":1.8919999999999999,"5min":2.3900000000000001,"15min":12.852},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.38800000000000001,"15min":0.48199999999999998},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.192},"max":{"1min":1.9410000000000001,"5min":1.9410000000000001,"15min":12.784000000000001},"last":0.22900000000000001}]},{"osd":1,"last update":"Tue Oct 17 02:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.496,"15min":0.47199999999999998},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.498,"5min":4.6980000000000004,"15min":6.202},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.56499999999999995,"15min":0.58599999999999997},"min":{"1min":0.26900000000000002,"5min":0.23999999999999999,"15min":0.23200000000000001},"max":{"1min":2.819,"5min":5.1779999999999999,"15min":12.824999999999999},"last":0.76100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739458,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5940916,"kb_used_data":4456684,"kb_used_omap":423,"kb_used_meta":1483800,"kb_avail":88430924,"statfs":{"total":96636764160,"available":90553266176,"internally_reserved":0,"allocated":4563644416,"data_stored":5401197417,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":434080,"internal_metadata":1519411296},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,3,4,6,18],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:12:33 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.61699999999999999,"15min":0.58799999999999997},"min":{"1min":0.17999999999999999,"5min":0.17100000000000001,"15min":0.16300000000000001},"max":{"1min":2.4750000000000001,"5min":4.2480000000000002,"15min":11.891999999999999},"last":0.312},{"interface":"front","average":{"1min":0.62,"5min":0.66200000000000003,"15min":0.63500000000000001},"min":{"1min":0.24099999999999999,"5min":0.224,"15min":0.22},"max":{"1min":2.5230000000000001,"5min":4.2869999999999999,"15min":11.926},"last":0.34899999999999998}]},{"osd":2,"last update":"Tue Oct 17 02:12:33 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.622,"15min":0.66500000000000004},"min":{"1min":0.20899999999999999,"5min":0.20300000000000001,"15min":0.19500000000000001},"max":{"1min":0.69399999999999995,"5min":6.3680000000000003,"15min":25.128},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.441,"15min":0.45700000000000002},"min":{"1min":0.24199999999999999,"5min":0.214,"15min":0.20100000000000001},"max":{"1min":0.71299999999999997,"5min":5.2709999999999999,"15min":6.7800000000000002},"last":0.29499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739459,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6219072,"kb_used_data":4597432,"kb_used_omap":454,"kb_used_meta":1621177,"kb_avail":88152768,"statfs":{"total":96636764160,"available":90268434432,"internally_reserved":0,"allocated":4707770368,"data_stored":5570168880,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":465730,"internal_metadata":1660085438},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,2,2,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:12:13 2023","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":0.64500000000000002,"15min":0.54100000000000004},"min":{"1min":0.19500000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":13.834,"5min":13.834,"15min":13.834},"last":0.20000000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":0.64900000000000002,"15min":0.55800000000000005},"min":{"1min":0.23200000000000001,"5min":0.19600000000000001,"15min":0.191},"max":{"1min":9.0730000000000004,"5min":9.0730000000000004,"15min":9.0730000000000004},"last":0.22500000000000001}]},{"osd":2,"last update":"Tue Oct 17 02:12:24 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.55300000000000005,"15min":0.58099999999999996},"min":{"1min":0.20599999999999999,"5min":0.187,"15min":0.187},"max":{"1min":0.65300000000000002,"5min":12.382,"15min":12.382},"last":0.245},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.73799999999999999,"15min":0.64900000000000002},"min":{"1min":0.26700000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":2.085,"5min":12.561,"15min":12.561},"last":0.26400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7184,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4706963456,"data_stored":5569560576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4562657280,"data_stored":5400490003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9268158464,"data_stored":10968588307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:13:05.944 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:13:05.944 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:13:11.859 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:13:11.853+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T02:13:11.879 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:13:11.873+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T02:13:25.946 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:13:26.580 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:13:26.581 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:13:26.625 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2852,"stamp":"2023-10-17T02:13:25.651574+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28946131800,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2298948,"num_read_kb":8993161,"num_write":2271288,"num_write_kb":9084803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88419,"ondisk_log_size":88419,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22910880,"kb_used_data":18166020,"kb_used_omap":1731,"kb_used_meta":4743100,"kb_avail":260204640,"statfs":{"total":289910292480,"available":266449551360,"internally_reserved":0,"allocated":18602004480,"data_stored":22001735395,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1773277,"internal_metadata":4856934691},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,6,2,9,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19304448,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4579,"num_read_kb":18284,"num_write":4738,"num_write_kb":18952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001883"},"pg_stats":[{"pgid":"2.7","version":"115'15051","reported_seq":21092,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:13:17.568931+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:13:17.568931+0000","last_peered":"2023-10-17T02:13:17.568931+0000","last_clean":"2023-10-17T02:13:17.568931+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:13:17.568931+0000","last_undegraded":"2023-10-17T02:13:17.568931+0000","last_fullsized":"2023-10-17T02:13:17.568931+0000","mapping_epoch":13,"log_start":"87'5000","ondisk_log_start":"87'5000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"115'11337","reported_seq":17044,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:13:17.545194+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:13:17.545194+0000","last_peered":"2023-10-17T02:13:17.545194+0000","last_clean":"2023-10-17T02:13:17.545194+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:13:17.545194+0000","last_undegraded":"2023-10-17T02:13:17.545194+0000","last_fullsized":"2023-10-17T02:13:17.545194+0000","mapping_epoch":13,"log_start":"54'1300","ondisk_log_start":"54'1300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"115'7600","reported_seq":10716,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:13:17.542088+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:13:17.542088+0000","last_peered":"2023-10-17T02:13:17.542088+0000","last_clean":"2023-10-17T02:13:17.542088+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:13:17.542088+0000","last_undegraded":"2023-10-17T02:13:17.542088+0000","last_fullsized":"2023-10-17T02:13:17.542088+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":7600,"log_dups_size":0,"ondisk_log_size":7600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"115'11371","reported_seq":16488,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:13:17.551621+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:13:17.551621+0000","last_peered":"2023-10-17T02:13:17.551621+0000","last_clean":"2023-10-17T02:13:17.551621+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:13:17.551621+0000","last_undegraded":"2023-10-17T02:13:17.551621+0000","last_fullsized":"2023-10-17T02:13:17.551621+0000","mapping_epoch":13,"log_start":"54'1300","ondisk_log_start":"54'1300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"115'578868","reported_seq":1154360,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:13:25.158832+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:13:25.158832+0000","last_peered":"2023-10-17T02:13:25.158832+0000","last_clean":"2023-10-17T02:13:25.158832+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:13:25.158832+0000","last_undegraded":"2023-10-17T02:13:25.158832+0000","last_fullsized":"2023-10-17T02:13:25.158832+0000","mapping_epoch":16,"log_start":"115'568810","ondisk_log_start":"115'568810","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7339474944,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":576174,"num_read_kb":2271588,"num_write":576111,"num_write_kb":2304444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":276,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:17.493344+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:11:17.493344+0000","last_peered":"2023-10-17T02:11:17.493344+0000","last_clean":"2023-10-17T02:11:17.493344+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:11:17.493344+0000","last_undegraded":"2023-10-17T02:11:17.493344+0000","last_fullsized":"2023-10-17T02:11:17.493344+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"115'556273","reported_seq":1110101,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:13:20.811348+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:13:20.811348+0000","last_peered":"2023-10-17T02:13:20.811348+0000","last_clean":"2023-10-17T02:13:20.811348+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:13:20.811348+0000","last_undegraded":"2023-10-17T02:13:20.811348+0000","last_fullsized":"2023-10-17T02:13:20.811348+0000","mapping_epoch":16,"log_start":"115'546219","ondisk_log_start":"115'546219","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":7054864384,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":554503,"num_read_kb":2188164,"num_write":553623,"num_write_kb":2214492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":3081,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:13:21.027441+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:13:21.027441+0000","last_peered":"2023-10-17T02:13:21.027441+0000","last_clean":"2023-10-17T02:13:21.027441+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:13:21.027441+0000","last_undegraded":"2023-10-17T02:13:21.027441+0000","last_fullsized":"2023-10-17T02:13:21.027441+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"115'579623","reported_seq":1157077,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:13:25.138191+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:13:25.138191+0000","last_peered":"2023-10-17T02:13:25.138191+0000","last_clean":"2023-10-17T02:13:25.138191+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:13:25.138191+0000","last_undegraded":"2023-10-17T02:13:25.138191+0000","last_fullsized":"2023-10-17T02:13:25.138191+0000","mapping_epoch":16,"log_start":"115'569525","ondisk_log_start":"115'569525","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7355699200,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":578165,"num_read_kb":2280352,"num_write":576869,"num_write_kb":2307476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4615,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:13:21.033142+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:13:21.033142+0000","last_peered":"2023-10-17T02:13:21.033142+0000","last_clean":"2023-10-17T02:13:21.033142+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:13:21.033142+0000","last_undegraded":"2023-10-17T02:13:21.033142+0000","last_fullsized":"2023-10-17T02:13:21.033142+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"115'566399","reported_seq":1132034,"reported_epoch":115,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:13:20.811814+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:13:20.811814+0000","last_peered":"2023-10-17T02:13:20.811814+0000","last_clean":"2023-10-17T02:13:20.811814+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:13:20.811814+0000","last_undegraded":"2023-10-17T02:13:20.811814+0000","last_fullsized":"2023-10-17T02:13:20.811814+0000","mapping_epoch":16,"log_start":"115'556318","ondisk_log_start":"115'556318","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1939,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 715s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7195500563,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":566144,"num_read_kb":2232404,"num_write":564055,"num_write_kb":2256214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"115'11342","reported_seq":15823,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:13:17.606258+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:13:17.606258+0000","last_peered":"2023-10-17T02:13:17.606258+0000","last_clean":"2023-10-17T02:13:17.606258+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:13:17.606258+0000","last_undegraded":"2023-10-17T02:13:17.606258+0000","last_fullsized":"2023-10-17T02:13:17.606258+0000","mapping_epoch":13,"log_start":"54'1300","ondisk_log_start":"54'1300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":297,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:17.499716+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:11:17.499716+0000","last_peered":"2023-10-17T02:11:17.499716+0000","last_clean":"2023-10-17T02:11:17.499716+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:11:17.499716+0000","last_undegraded":"2023-10-17T02:11:17.499716+0000","last_fullsized":"2023-10-17T02:11:17.499716+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28945539091,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2274986,"num_read_kb":8972508,"num_write":2270658,"num_write_kb":9082626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18599612416,"data_stored":22000472102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40291,"ondisk_log_size":40291,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34223,"internal_metadata":0},"log_size":48029,"ondisk_log_size":48029,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739466,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10587108,"kb_used_data":9081688,"kb_used_omap":852,"kb_used_meta":1504555,"kb_avail":83784732,"statfs":{"total":96636764160,"available":85795565568,"internally_reserved":0,"allocated":9299648512,"data_stored":10999427914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":873467,"internal_metadata":1540664325},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:12:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.436,"15min":0.45900000000000002},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":1.8919999999999999,"5min":2.3900000000000001,"15min":12.852},"last":0.253},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.38800000000000001,"15min":0.48199999999999998},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.192},"max":{"1min":1.9410000000000001,"5min":1.9410000000000001,"15min":12.784000000000001},"last":0.46600000000000003}]},{"osd":1,"last update":"Tue Oct 17 02:13:10 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.46500000000000002,"15min":0.47499999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.20899999999999999},"max":{"1min":0.80600000000000005,"5min":4.0730000000000004,"15min":6.202},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.52400000000000002,"15min":0.58199999999999996},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23200000000000001},"max":{"1min":0.76100000000000001,"5min":5.1779999999999999,"15min":12.824999999999999},"last":0.33800000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739462,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6035196,"kb_used_data":4471540,"kb_used_omap":423,"kb_used_meta":1563224,"kb_avail":88336644,"statfs":{"total":96636764160,"available":90456723456,"internally_reserved":0,"allocated":4578856960,"data_stored":5416409961,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":434080,"internal_metadata":1600741472},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,1,2,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:12:33 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.61699999999999999,"15min":0.58799999999999997},"min":{"1min":0.17999999999999999,"5min":0.17100000000000001,"15min":0.16300000000000001},"max":{"1min":2.4750000000000001,"5min":4.2480000000000002,"15min":11.891999999999999},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.62,"5min":0.66200000000000003,"15min":0.63500000000000001},"min":{"1min":0.24099999999999999,"5min":0.224,"15min":0.22},"max":{"1min":2.5230000000000001,"5min":4.2869999999999999,"15min":11.926},"last":0.25700000000000001}]},{"osd":2,"last update":"Tue Oct 17 02:12:33 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.622,"15min":0.66500000000000004},"min":{"1min":0.20899999999999999,"5min":0.20300000000000001,"15min":0.19500000000000001},"max":{"1min":0.69399999999999995,"5min":6.3680000000000003,"15min":25.128},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.441,"15min":0.45700000000000002},"min":{"1min":0.24199999999999999,"5min":0.214,"15min":0.20100000000000001},"max":{"1min":0.71299999999999997,"5min":5.2709999999999999,"15min":6.7800000000000002},"last":0.45300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739463,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6288576,"kb_used_data":4612792,"kb_used_omap":454,"kb_used_meta":1675321,"kb_avail":88083264,"statfs":{"total":96636764160,"available":90197262336,"internally_reserved":0,"allocated":4723499008,"data_stored":5585897520,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":465730,"internal_metadata":1715528894},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:13:16 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.65000000000000002,"15min":0.55500000000000005},"min":{"1min":0.184,"5min":0.184,"15min":0.16800000000000001},"max":{"1min":3.9100000000000001,"5min":13.834,"15min":13.834},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.61799999999999999,"15min":0.55000000000000004},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.191},"max":{"1min":1.6259999999999999,"5min":9.0730000000000004,"15min":9.0730000000000004},"last":0.215}]},{"osd":2,"last update":"Tue Oct 17 02:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.54100000000000004,"15min":0.58799999999999997},"min":{"1min":0.20799999999999999,"5min":0.20599999999999999,"15min":0.187},"max":{"1min":3.1709999999999998,"5min":12.382,"15min":12.382},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.308,"5min":0.67800000000000005,"15min":0.61599999999999999},"min":{"1min":0.22600000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.54000000000000004,"5min":12.561,"15min":12.561},"last":0.26100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7184,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4722606080,"data_stored":5585203200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4578013184,"data_stored":5415845907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9298993152,"data_stored":10999422995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:13:26.627 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:13:26.628 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-17T02:13:27.238 INFO:teuthology.orchestra.run.smithi169.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-17T02:13:27.271 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:13:47.272 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:13:47.815 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:13:47.816 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:13:47.849 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2863,"stamp":"2023-10-17T02:13:47.657787+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28980919128,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2307148,"num_read_kb":9025897,"num_write":2279803,"num_write_kb":9118863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88384,"ondisk_log_size":88384,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23276004,"kb_used_data":18237452,"kb_used_omap":1731,"kb_used_meta":5036796,"kb_avail":259839516,"statfs":{"total":289910292480,"available":266075664384,"internally_reserved":0,"allocated":18675150848,"data_stored":22074881763,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1773277,"internal_metadata":5157679395},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,7,11,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19542016,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4512,"num_read_kb":18000,"num_write":4775,"num_write_kb":19100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004372"},"pg_stats":[{"pgid":"2.7","version":"115'15111","reported_seq":21173,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:13:44.245887+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:13:44.245887+0000","last_peered":"2023-10-17T02:13:44.245887+0000","last_clean":"2023-10-17T02:13:44.245887+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:13:44.245887+0000","last_undegraded":"2023-10-17T02:13:44.245887+0000","last_fullsized":"2023-10-17T02:13:44.245887+0000","mapping_epoch":13,"log_start":"87'5100","ondisk_log_start":"87'5100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"115'11382","reported_seq":17105,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:13:44.239068+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:13:44.239068+0000","last_peered":"2023-10-17T02:13:44.239068+0000","last_clean":"2023-10-17T02:13:44.239068+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:13:44.239068+0000","last_undegraded":"2023-10-17T02:13:44.239068+0000","last_fullsized":"2023-10-17T02:13:44.239068+0000","mapping_epoch":13,"log_start":"54'1300","ondisk_log_start":"54'1300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"115'7630","reported_seq":10756,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:13:44.215205+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:13:44.215205+0000","last_peered":"2023-10-17T02:13:44.215205+0000","last_clean":"2023-10-17T02:13:44.215205+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:13:44.215205+0000","last_undegraded":"2023-10-17T02:13:44.215205+0000","last_fullsized":"2023-10-17T02:13:44.215205+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":7630,"log_dups_size":0,"ondisk_log_size":7630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"115'11416","reported_seq":16548,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:13:44.242765+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:13:44.242765+0000","last_peered":"2023-10-17T02:13:44.242765+0000","last_clean":"2023-10-17T02:13:44.242765+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:13:44.242765+0000","last_undegraded":"2023-10-17T02:13:44.242765+0000","last_fullsized":"2023-10-17T02:13:44.242765+0000","mapping_epoch":13,"log_start":"54'1400","ondisk_log_start":"54'1400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"115'580769","reported_seq":1158062,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:13:45.288020+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:13:45.288020+0000","last_peered":"2023-10-17T02:13:45.288020+0000","last_clean":"2023-10-17T02:13:45.288020+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:13:45.288020+0000","last_undegraded":"2023-10-17T02:13:45.288020+0000","last_fullsized":"2023-10-17T02:13:45.288020+0000","mapping_epoch":16,"log_start":"115'570710","ondisk_log_start":"115'570710","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7347273728,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":577971,"num_read_kb":2278764,"num_write":578012,"num_write_kb":2312048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":276,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:17.493344+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:11:17.493344+0000","last_peered":"2023-10-17T02:11:17.493344+0000","last_clean":"2023-10-17T02:11:17.493344+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:11:17.493344+0000","last_undegraded":"2023-10-17T02:11:17.493344+0000","last_fullsized":"2023-10-17T02:11:17.493344+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"115'558656","reported_seq":1114822,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:13:46.005002+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:13:46.005002+0000","last_peered":"2023-10-17T02:13:46.005002+0000","last_clean":"2023-10-17T02:13:46.005002+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:13:46.005002+0000","last_undegraded":"2023-10-17T02:13:46.005002+0000","last_fullsized":"2023-10-17T02:13:46.005002+0000","mapping_epoch":16,"log_start":"115'548619","ondisk_log_start":"115'548619","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":7064592384,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":556840,"num_read_kb":2197488,"num_write":556006,"num_write_kb":2224024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":3091,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:13:46.699490+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:13:46.699490+0000","last_peered":"2023-10-17T02:13:46.699490+0000","last_clean":"2023-10-17T02:13:46.699490+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:13:46.699490+0000","last_undegraded":"2023-10-17T02:13:46.699490+0000","last_fullsized":"2023-10-17T02:13:46.699490+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"115'581493","reported_seq":1160803,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:13:45.270599+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:13:45.270599+0000","last_peered":"2023-10-17T02:13:45.270599+0000","last_clean":"2023-10-17T02:13:45.270599+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:13:45.270599+0000","last_undegraded":"2023-10-17T02:13:45.270599+0000","last_fullsized":"2023-10-17T02:13:45.270599+0000","mapping_epoch":16,"log_start":"115'571425","ondisk_log_start":"115'571425","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7363334144,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":580019,"num_read_kb":2287748,"num_write":578739,"num_write_kb":2314956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4630,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:13:46.628175+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:13:46.628175+0000","last_peered":"2023-10-17T02:13:46.628175+0000","last_clean":"2023-10-17T02:13:46.628175+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:13:46.628175+0000","last_undegraded":"2023-10-17T02:13:46.628175+0000","last_fullsized":"2023-10-17T02:13:46.628175+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"115'568785","reported_seq":1136636,"reported_epoch":115,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:13:46.023869+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:13:46.023869+0000","last_peered":"2023-10-17T02:13:46.023869+0000","last_clean":"2023-10-17T02:13:46.023869+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:13:46.023869+0000","last_undegraded":"2023-10-17T02:13:46.023869+0000","last_fullsized":"2023-10-17T02:13:46.023869+0000","mapping_epoch":16,"log_start":"115'558718","ondisk_log_start":"115'558718","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":1987,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 740s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7205126163,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":568356,"num_read_kb":2241244,"num_write":566416,"num_write_kb":2265658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"115'11387","reported_seq":15883,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:13:44.238756+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:13:44.238756+0000","last_peered":"2023-10-17T02:13:44.238756+0000","last_clean":"2023-10-17T02:13:44.238756+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:13:44.238756+0000","last_undegraded":"2023-10-17T02:13:44.238756+0000","last_fullsized":"2023-10-17T02:13:44.238756+0000","mapping_epoch":13,"log_start":"54'1300","ondisk_log_start":"54'1300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":297,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:17.499716+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:11:17.499716+0000","last_peered":"2023-10-17T02:11:17.499716+0000","last_clean":"2023-10-17T02:11:17.499716+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:11:17.499716+0000","last_undegraded":"2023-10-17T02:11:17.499716+0000","last_fullsized":"2023-10-17T02:11:17.499716+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28980326419,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2283186,"num_read_kb":9005244,"num_write":2279173,"num_write_kb":9116686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18673389568,"data_stored":22074249254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40231,"ondisk_log_size":40231,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34223,"internal_metadata":0},"log_size":48054,"ondisk_log_size":48054,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739471,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10780600,"kb_used_data":9119408,"kb_used_omap":852,"kb_used_meta":1660331,"kb_avail":83591240,"statfs":{"total":96636764160,"available":85597429760,"internally_reserved":0,"allocated":9338273792,"data_stored":11038053194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":873467,"internal_metadata":1700178949},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:13:39 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.433,"15min":0.45800000000000002},"min":{"1min":0.17799999999999999,"5min":0.158,"15min":0.158},"max":{"1min":1.853,"5min":2.3900000000000001,"15min":12.852},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40600000000000003,"15min":0.47899999999999998},"min":{"1min":0.21299999999999999,"5min":0.20300000000000001,"15min":0.192},"max":{"1min":1.744,"5min":1.9410000000000001,"15min":12.784000000000001},"last":0.36699999999999999}]},{"osd":1,"last update":"Tue Oct 17 02:13:10 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.46500000000000002,"15min":0.47499999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.20899999999999999},"max":{"1min":0.80600000000000005,"5min":4.0730000000000004,"15min":6.202},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.52400000000000002,"15min":0.58199999999999996},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23200000000000001},"max":{"1min":0.76100000000000001,"5min":5.1779999999999999,"15min":12.824999999999999},"last":0.30499999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739467,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6112264,"kb_used_data":4490304,"kb_used_omap":423,"kb_used_meta":1621528,"kb_avail":88259576,"statfs":{"total":96636764160,"available":90377805824,"internally_reserved":0,"allocated":4598071296,"data_stored":5435624297,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":434080,"internal_metadata":1660444768},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,2,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:13:33 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.66500000000000004,"15min":0.62},"min":{"1min":0.192,"5min":0.17100000000000001,"15min":0.16600000000000001},"max":{"1min":5.1109999999999998,"5min":5.1109999999999998,"15min":11.891999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.71199999999999997,"15min":0.66700000000000004},"min":{"1min":0.25700000000000001,"5min":0.224,"15min":0.224},"max":{"1min":5.1449999999999996,"5min":5.1449999999999996,"15min":11.926},"last":0.73299999999999998}]},{"osd":2,"last update":"Tue Oct 17 02:13:33 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.61899999999999999,"15min":0.67300000000000004},"min":{"1min":0.253,"5min":0.20599999999999999,"15min":0.19500000000000001},"max":{"1min":3.4550000000000001,"5min":6.3680000000000003,"15min":25.128},"last":0.313},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.46800000000000003,"15min":0.47299999999999998},"min":{"1min":0.24399999999999999,"5min":0.219,"15min":0.214},"max":{"1min":3.4860000000000002,"5min":5.2709999999999999,"15min":6.7800000000000002},"last":0.67200000000000004}]}]},{"osd":0,"up_from":8,"seq":34359739467,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6383140,"kb_used_data":4627740,"kb_used_omap":454,"kb_used_meta":1754937,"kb_avail":87988700,"statfs":{"total":96636764160,"available":90100428800,"internally_reserved":0,"allocated":4738805760,"data_stored":5601204272,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":465730,"internal_metadata":1797055678},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,2,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:13:16 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.65000000000000002,"15min":0.55500000000000005},"min":{"1min":0.184,"5min":0.184,"15min":0.16800000000000001},"max":{"1min":3.9100000000000001,"5min":13.834,"15min":13.834},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.61799999999999999,"15min":0.55000000000000004},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.191},"max":{"1min":1.6259999999999999,"5min":9.0730000000000004,"15min":9.0730000000000004},"last":0.35499999999999998}]},{"osd":2,"last update":"Tue Oct 17 02:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.54100000000000004,"15min":0.58799999999999997},"min":{"1min":0.20799999999999999,"5min":0.20599999999999999,"15min":0.187},"max":{"1min":3.1709999999999998,"5min":12.382,"15min":12.382},"last":0.246},{"interface":"front","average":{"1min":0.308,"5min":0.67800000000000005,"15min":0.61599999999999999},"min":{"1min":0.22600000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.54000000000000004,"5min":12.561,"15min":12.561},"last":0.32100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7184,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4738080768,"data_stored":5600677888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4597440512,"data_stored":5435273235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9337868288,"data_stored":11038298131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:13:47.852 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:13:47.852 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:13:48.768 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:13:48.761+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T02:13:48.768 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:13:48.761+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T02:13:48.769 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:13:48.761+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbda750 2023-10-17T02:13:48.769 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:13:48.761+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T02:13:48.769 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:13:48.761+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T02:13:48.769 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:13:48.761+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T02:13:48.770 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:13:48.761+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T02:13:48.770 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:13:48.761+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T02:13:48.771 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:13:48.761+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T02:13:48.771 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:13:48.761+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T02:13:48.771 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:13:48.761+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T02:13:48.772 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:13:48.761+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T02:13:48.772 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:13:48.761+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T02:13:48.773 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:13:48.761+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T02:13:48.773 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:13:48.761+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T02:14:07.854 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:14:08.396 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:14:08.396 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:14:08.447 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2873,"stamp":"2023-10-17T02:14:07.661327+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29011848024,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2314775,"num_read_kb":9056341,"num_write":2287349,"num_write_kb":9149047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88381,"ondisk_log_size":88381,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23552100,"kb_used_data":18297804,"kb_used_omap":1731,"kb_used_meta":5252540,"kb_avail":259563420,"statfs":{"total":289910292480,"available":265792942080,"internally_reserved":0,"allocated":18736951296,"data_stored":22136682211,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1773277,"internal_metadata":5378601251},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,4,4,7,17,21,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19517440,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4837,"num_read_kb":19304,"num_write":4770,"num_write_kb":19080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002068"},"pg_stats":[{"pgid":"2.7","version":"115'15143","reported_seq":21221,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:04.084048+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:14:04.084048+0000","last_peered":"2023-10-17T02:14:04.084048+0000","last_clean":"2023-10-17T02:14:04.084048+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:14:04.084048+0000","last_undegraded":"2023-10-17T02:14:04.084048+0000","last_fullsized":"2023-10-17T02:14:04.084048+0000","mapping_epoch":13,"log_start":"87'5100","ondisk_log_start":"87'5100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"115'11406","reported_seq":17141,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:04.085007+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:14:04.085007+0000","last_peered":"2023-10-17T02:14:04.085007+0000","last_clean":"2023-10-17T02:14:04.085007+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:14:04.085007+0000","last_undegraded":"2023-10-17T02:14:04.085007+0000","last_fullsized":"2023-10-17T02:14:04.085007+0000","mapping_epoch":13,"log_start":"54'1400","ondisk_log_start":"54'1400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"115'7646","reported_seq":10780,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:04.074431+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:14:04.074431+0000","last_peered":"2023-10-17T02:14:04.074431+0000","last_clean":"2023-10-17T02:14:04.074431+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:14:04.074431+0000","last_undegraded":"2023-10-17T02:14:04.074431+0000","last_fullsized":"2023-10-17T02:14:04.074431+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":7646,"log_dups_size":0,"ondisk_log_size":7646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"115'11440","reported_seq":16584,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:04.095959+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:14:04.095959+0000","last_peered":"2023-10-17T02:14:04.095959+0000","last_clean":"2023-10-17T02:14:04.095959+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:14:04.095959+0000","last_undegraded":"2023-10-17T02:14:04.095959+0000","last_fullsized":"2023-10-17T02:14:04.095959+0000","mapping_epoch":13,"log_start":"54'1400","ondisk_log_start":"54'1400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"115'582632","reported_seq":1161824,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:05.410500+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:14:05.410500+0000","last_peered":"2023-10-17T02:14:05.410500+0000","last_clean":"2023-10-17T02:14:05.410500+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:14:05.410500+0000","last_undegraded":"2023-10-17T02:14:05.410500+0000","last_fullsized":"2023-10-17T02:14:05.410500+0000","mapping_epoch":16,"log_start":"115'572610","ondisk_log_start":"115'572610","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7354888192,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":579870,"num_read_kb":2286340,"num_write":579875,"num_write_kb":2319500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":276,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:17.493344+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:11:17.493344+0000","last_peered":"2023-10-17T02:11:17.493344+0000","last_clean":"2023-10-17T02:11:17.493344+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:11:17.493344+0000","last_undegraded":"2023-10-17T02:11:17.493344+0000","last_fullsized":"2023-10-17T02:11:17.493344+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"115'560519","reported_seq":1118566,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:06.173242+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:14:06.173242+0000","last_peered":"2023-10-17T02:14:06.173242+0000","last_clean":"2023-10-17T02:14:06.173242+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:14:06.173242+0000","last_undegraded":"2023-10-17T02:14:06.173242+0000","last_fullsized":"2023-10-17T02:14:06.173242+0000","mapping_epoch":16,"log_start":"115'550419","ondisk_log_start":"115'550419","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":7072198656,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":558721,"num_read_kb":2204996,"num_write":557869,"num_write_kb":2231476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":3099,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:07.106248+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:14:07.106248+0000","last_peered":"2023-10-17T02:14:07.106248+0000","last_clean":"2023-10-17T02:14:07.106248+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:14:07.106248+0000","last_undegraded":"2023-10-17T02:14:07.106248+0000","last_fullsized":"2023-10-17T02:14:07.106248+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"115'583425","reported_seq":1164664,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:05.397345+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:14:05.397345+0000","last_peered":"2023-10-17T02:14:05.397345+0000","last_clean":"2023-10-17T02:14:05.397345+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:14:05.397345+0000","last_undegraded":"2023-10-17T02:14:05.397345+0000","last_fullsized":"2023-10-17T02:14:05.397345+0000","mapping_epoch":16,"log_start":"115'573325","ondisk_log_start":"115'573325","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7371296768,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":581949,"num_read_kb":2295448,"num_write":580671,"num_write_kb":2322684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4642,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:07.111049+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:14:07.111049+0000","last_peered":"2023-10-17T02:14:07.111049+0000","last_clean":"2023-10-17T02:14:07.111049+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:14:07.111049+0000","last_undegraded":"2023-10-17T02:14:07.111049+0000","last_fullsized":"2023-10-17T02:14:07.111049+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"115'570704","reported_seq":1140479,"reported_epoch":115,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:14:06.204359+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:14:06.204359+0000","last_peered":"2023-10-17T02:14:06.204359+0000","last_clean":"2023-10-17T02:14:06.204359+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:14:06.204359+0000","last_undegraded":"2023-10-17T02:14:06.204359+0000","last_fullsized":"2023-10-17T02:14:06.204359+0000","mapping_epoch":16,"log_start":"115'560618","ondisk_log_start":"115'560618","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2047,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 760s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7212871699,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":570273,"num_read_kb":2248904,"num_write":568304,"num_write_kb":2273210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"115'11411","reported_seq":15919,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:04.122991+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:14:04.122991+0000","last_peered":"2023-10-17T02:14:04.122991+0000","last_clean":"2023-10-17T02:14:04.122991+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:14:04.122991+0000","last_undegraded":"2023-10-17T02:14:04.122991+0000","last_fullsized":"2023-10-17T02:14:04.122991+0000","mapping_epoch":13,"log_start":"55'1400","ondisk_log_start":"55'1400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":297,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:17.499716+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:11:17.499716+0000","last_peered":"2023-10-17T02:11:17.499716+0000","last_clean":"2023-10-17T02:11:17.499716+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:11:17.499716+0000","last_undegraded":"2023-10-17T02:11:17.499716+0000","last_fullsized":"2023-10-17T02:11:17.499716+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29011255315,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2290813,"num_read_kb":9035688,"num_write":2286719,"num_write_kb":9146870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18735144960,"data_stored":22136004646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40308,"ondisk_log_size":40308,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34223,"internal_metadata":0},"log_size":47974,"ondisk_log_size":47974,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739475,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10896580,"kb_used_data":9149308,"kb_used_omap":852,"kb_used_meta":1746411,"kb_avail":83475260,"statfs":{"total":96636764160,"available":85478666240,"internally_reserved":0,"allocated":9368891392,"data_stored":11068670794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":873467,"internal_metadata":1788324869},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,3,6,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:13:39 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.433,"15min":0.45800000000000002},"min":{"1min":0.17799999999999999,"5min":0.158,"15min":0.158},"max":{"1min":1.853,"5min":2.3900000000000001,"15min":12.852},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40600000000000003,"15min":0.47899999999999998},"min":{"1min":0.21299999999999999,"5min":0.20300000000000001,"15min":0.192},"max":{"1min":1.744,"5min":1.9410000000000001,"15min":12.784000000000001},"last":0.47399999999999998}]},{"osd":1,"last update":"Tue Oct 17 02:13:10 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.46500000000000002,"15min":0.47499999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.20899999999999999},"max":{"1min":0.80600000000000005,"5min":4.0730000000000004,"15min":6.202},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.52400000000000002,"15min":0.58199999999999996},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23200000000000001},"max":{"1min":0.76100000000000001,"5min":5.1779999999999999,"15min":12.824999999999999},"last":0.26000000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739471,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6181552,"kb_used_data":4505448,"kb_used_omap":423,"kb_used_meta":1675672,"kb_avail":88190288,"statfs":{"total":96636764160,"available":90306854912,"internally_reserved":0,"allocated":4613578752,"data_stored":5451131753,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":434080,"internal_metadata":1715888224},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,8,19,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:13:33 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.66500000000000004,"15min":0.62},"min":{"1min":0.192,"5min":0.17100000000000001,"15min":0.16600000000000001},"max":{"1min":5.1109999999999998,"5min":5.1109999999999998,"15min":11.891999999999999},"last":1.1779999999999999},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.71199999999999997,"15min":0.66700000000000004},"min":{"1min":0.25700000000000001,"5min":0.224,"15min":0.224},"max":{"1min":5.1449999999999996,"5min":5.1449999999999996,"15min":11.926},"last":1.2210000000000001}]},{"osd":2,"last update":"Tue Oct 17 02:13:33 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.61899999999999999,"15min":0.67300000000000004},"min":{"1min":0.253,"5min":0.20599999999999999,"15min":0.19500000000000001},"max":{"1min":3.4550000000000001,"5min":6.3680000000000003,"15min":25.128},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.46800000000000003,"15min":0.47299999999999998},"min":{"1min":0.24399999999999999,"5min":0.219,"15min":0.214},"max":{"1min":3.4860000000000002,"5min":5.2709999999999999,"15min":6.7800000000000002},"last":0.46000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739471,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6473968,"kb_used_data":4643048,"kb_used_omap":454,"kb_used_meta":1830457,"kb_avail":87897872,"statfs":{"total":96636764160,"available":90007420928,"internally_reserved":0,"allocated":4754481152,"data_stored":5616879664,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":465730,"internal_metadata":1874388158},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:13:16 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.65000000000000002,"15min":0.55500000000000005},"min":{"1min":0.184,"5min":0.184,"15min":0.16800000000000001},"max":{"1min":3.9100000000000001,"5min":13.834,"15min":13.834},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.61799999999999999,"15min":0.55000000000000004},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.191},"max":{"1min":1.6259999999999999,"5min":9.0730000000000004,"15min":9.0730000000000004},"last":0.35799999999999998}]},{"osd":2,"last update":"Tue Oct 17 02:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.54100000000000004,"15min":0.58799999999999997},"min":{"1min":0.20799999999999999,"5min":0.20599999999999999,"15min":0.187},"max":{"1min":3.1709999999999998,"5min":12.382,"15min":12.382},"last":0.442},{"interface":"front","average":{"1min":0.308,"5min":0.67800000000000005,"15min":0.61599999999999999},"min":{"1min":0.22600000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.54000000000000004,"5min":12.561,"15min":12.561},"last":0.32100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7184,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4753616896,"data_stored":5616214016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4612816896,"data_stored":5450649619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9368711168,"data_stored":11069141011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:14:08.449 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:14:08.449 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:14:08.802 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:14:08.796+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T02:14:08.811 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:14:08.804+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T02:14:08.829 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:14:08.824+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc784e0 2023-10-17T02:14:08.833 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:14:08.828+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T02:14:08.833 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:14:08.828+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T02:14:08.833 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:14:08.828+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T02:14:08.834 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:14:08.828+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T02:14:08.834 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:14:08.828+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T02:14:08.834 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:14:08.828+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T02:14:08.834 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:14:08.828+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T02:14:08.835 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:14:08.828+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T02:14:08.835 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:14:08.828+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71040 2023-10-17T02:14:08.835 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:14:08.828+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a71930 2023-10-17T02:14:08.836 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:14:08.828+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T02:14:14.780 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:14:14.776+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T02:14:14.780 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:14:14.776+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T02:14:14.808 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:14:14.804+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T02:14:28.451 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:14:29.032 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:14:29.033 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:14:29.068 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2883,"stamp":"2023-10-17T02:14:27.664790+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29042654040,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2322297,"num_read_kb":9086337,"num_write":2294923,"num_write_kb":9179343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88333,"ondisk_log_size":88333,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23812192,"kb_used_data":18357576,"kb_used_omap":1731,"kb_used_meta":5452860,"kb_avail":259303328,"statfs":{"total":289910292480,"available":265526607872,"internally_reserved":0,"allocated":18798157824,"data_stored":22197888739,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1773277,"internal_metadata":5583728931},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,14,6,7,15,14,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19419136,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4698,"num_read_kb":18748,"num_write":4775,"num_write_kb":19100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002122"},"pg_stats":[{"pgid":"2.7","version":"115'15183","reported_seq":21277,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:22.464333+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:14:22.464333+0000","last_peered":"2023-10-17T02:14:22.464333+0000","last_clean":"2023-10-17T02:14:22.464333+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:14:22.464333+0000","last_undegraded":"2023-10-17T02:14:22.464333+0000","last_fullsized":"2023-10-17T02:14:22.464333+0000","mapping_epoch":13,"log_start":"87'5100","ondisk_log_start":"87'5100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"115'11436","reported_seq":17183,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:22.463813+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:14:22.463813+0000","last_peered":"2023-10-17T02:14:22.463813+0000","last_clean":"2023-10-17T02:14:22.463813+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:14:22.463813+0000","last_undegraded":"2023-10-17T02:14:22.463813+0000","last_fullsized":"2023-10-17T02:14:22.463813+0000","mapping_epoch":13,"log_start":"54'1400","ondisk_log_start":"54'1400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"115'7667","reported_seq":10810,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:22.464059+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:14:22.464059+0000","last_peered":"2023-10-17T02:14:22.464059+0000","last_clean":"2023-10-17T02:14:22.464059+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:14:22.464059+0000","last_undegraded":"2023-10-17T02:14:22.464059+0000","last_fullsized":"2023-10-17T02:14:22.464059+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":7667,"log_dups_size":0,"ondisk_log_size":7667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"115'11471","reported_seq":16628,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:22.463856+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:14:22.463856+0000","last_peered":"2023-10-17T02:14:22.463856+0000","last_clean":"2023-10-17T02:14:22.463856+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:14:22.463856+0000","last_undegraded":"2023-10-17T02:14:22.463856+0000","last_fullsized":"2023-10-17T02:14:22.463856+0000","mapping_epoch":13,"log_start":"54'1400","ondisk_log_start":"54'1400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"115'584545","reported_seq":1165680,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:25.469636+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:14:25.469636+0000","last_peered":"2023-10-17T02:14:25.469636+0000","last_clean":"2023-10-17T02:14:25.469636+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:14:25.469636+0000","last_undegraded":"2023-10-17T02:14:25.469636+0000","last_fullsized":"2023-10-17T02:14:25.469636+0000","mapping_epoch":16,"log_start":"115'574510","ondisk_log_start":"115'574510","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7362695168,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":581812,"num_read_kb":2294068,"num_write":581788,"num_write_kb":2327152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":277,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:12:19.777785+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:12:19.777785+0000","last_peered":"2023-10-17T02:12:19.777785+0000","last_clean":"2023-10-17T02:12:19.777785+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:12:19.777785+0000","last_undegraded":"2023-10-17T02:12:19.777785+0000","last_fullsized":"2023-10-17T02:12:19.777785+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"115'562332","reported_seq":1122208,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:26.293542+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:14:26.293542+0000","last_peered":"2023-10-17T02:14:26.293542+0000","last_clean":"2023-10-17T02:14:26.293542+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:14:26.293542+0000","last_undegraded":"2023-10-17T02:14:26.293542+0000","last_fullsized":"2023-10-17T02:14:26.293542+0000","mapping_epoch":16,"log_start":"115'552319","ondisk_log_start":"115'552319","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":7079542784,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":560549,"num_read_kb":2212296,"num_write":559682,"num_write_kb":2238728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":3106,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:22.469465+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:14:22.469465+0000","last_peered":"2023-10-17T02:14:22.469465+0000","last_clean":"2023-10-17T02:14:22.469465+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:14:22.469465+0000","last_undegraded":"2023-10-17T02:14:22.469465+0000","last_fullsized":"2023-10-17T02:14:22.469465+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"115'585376","reported_seq":1168562,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:25.465850+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:14:25.465850+0000","last_peered":"2023-10-17T02:14:25.465850+0000","last_clean":"2023-10-17T02:14:25.465850+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:14:25.465850+0000","last_undegraded":"2023-10-17T02:14:25.465850+0000","last_fullsized":"2023-10-17T02:14:25.465850+0000","mapping_epoch":16,"log_start":"115'575325","ondisk_log_start":"115'575325","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7379214336,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":583899,"num_read_kb":2303232,"num_write":582622,"num_write_kb":2330488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4651,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:22.465942+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:14:22.465942+0000","last_peered":"2023-10-17T02:14:22.465942+0000","last_clean":"2023-10-17T02:14:22.465942+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:14:22.465942+0000","last_undegraded":"2023-10-17T02:14:22.465942+0000","last_fullsized":"2023-10-17T02:14:22.465942+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"115'572626","reported_seq":1144215,"reported_epoch":115,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:14:26.306286+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:14:26.306286+0000","last_peered":"2023-10-17T02:14:26.306286+0000","last_clean":"2023-10-17T02:14:26.306286+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:14:26.306286+0000","last_undegraded":"2023-10-17T02:14:26.306286+0000","last_fullsized":"2023-10-17T02:14:26.306286+0000","mapping_epoch":16,"log_start":"115'562618","ondisk_log_start":"115'562618","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2093,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 780s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7220609043,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":572075,"num_read_kb":2256088,"num_write":570201,"num_write_kb":2280798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"115'11442","reported_seq":15963,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:22.463796+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:14:22.463796+0000","last_peered":"2023-10-17T02:14:22.463796+0000","last_clean":"2023-10-17T02:14:22.463796+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:14:22.463796+0000","last_undegraded":"2023-10-17T02:14:22.463796+0000","last_fullsized":"2023-10-17T02:14:22.463796+0000","mapping_epoch":13,"log_start":"55'1400","ondisk_log_start":"55'1400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":297,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:11:17.499716+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:11:17.499716+0000","last_peered":"2023-10-17T02:11:17.499716+0000","last_clean":"2023-10-17T02:11:17.499716+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:11:17.499716+0000","last_undegraded":"2023-10-17T02:11:17.499716+0000","last_fullsized":"2023-10-17T02:11:17.499716+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29042061331,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2298335,"num_read_kb":9065684,"num_write":2294293,"num_write_kb":9177166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18797154304,"data_stored":22198013990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40107,"ondisk_log_size":40107,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34223,"internal_metadata":0},"log_size":48127,"ondisk_log_size":48127,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739479,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11040860,"kb_used_data":9179348,"kb_used_omap":852,"kb_used_meta":1860651,"kb_avail":83330980,"statfs":{"total":96636764160,"available":85330923520,"internally_reserved":0,"allocated":9399652352,"data_stored":11099431754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":873467,"internal_metadata":1905306629},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:13:39 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.433,"15min":0.45800000000000002},"min":{"1min":0.17799999999999999,"5min":0.158,"15min":0.158},"max":{"1min":1.853,"5min":2.3900000000000001,"15min":12.852},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40600000000000003,"15min":0.47899999999999998},"min":{"1min":0.21299999999999999,"5min":0.20300000000000001,"15min":0.192},"max":{"1min":1.744,"5min":1.9410000000000001,"15min":12.784000000000001},"last":0.23599999999999999}]},{"osd":1,"last update":"Tue Oct 17 02:14:13 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.47299999999999998,"15min":0.44500000000000001},"min":{"1min":0.215,"5min":0.215,"15min":0.20899999999999999},"max":{"1min":1.228,"5min":4.0730000000000004,"15min":6.202},"last":0.314},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.495,"15min":0.55200000000000005},"min":{"1min":0.25600000000000001,"5min":0.23899999999999999,"15min":0.23200000000000001},"max":{"1min":1.115,"5min":5.1779999999999999,"15min":12.824999999999999},"last":0.34999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739475,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6246160,"kb_used_data":4520072,"kb_used_omap":423,"kb_used_meta":1725656,"kb_avail":88125680,"statfs":{"total":96636764160,"available":90240696320,"internally_reserved":0,"allocated":4628553728,"data_stored":5466106729,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":434080,"internal_metadata":1767071840},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,9,0,6,6,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:13:33 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.66500000000000004,"15min":0.62},"min":{"1min":0.192,"5min":0.17100000000000001,"15min":0.16600000000000001},"max":{"1min":5.1109999999999998,"5min":5.1109999999999998,"15min":11.891999999999999},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.71199999999999997,"15min":0.66700000000000004},"min":{"1min":0.25700000000000001,"5min":0.224,"15min":0.224},"max":{"1min":5.1449999999999996,"5min":5.1449999999999996,"15min":11.926},"last":0.27300000000000002}]},{"osd":2,"last update":"Tue Oct 17 02:13:33 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.61899999999999999,"15min":0.67300000000000004},"min":{"1min":0.253,"5min":0.20599999999999999,"15min":0.19500000000000001},"max":{"1min":3.4550000000000001,"5min":6.3680000000000003,"15min":25.128},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.46800000000000003,"15min":0.47299999999999998},"min":{"1min":0.24399999999999999,"5min":0.219,"15min":0.214},"max":{"1min":3.4860000000000002,"5min":5.2709999999999999,"15min":6.7800000000000002},"last":0.23000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739475,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6525172,"kb_used_data":4658156,"kb_used_omap":454,"kb_used_meta":1866553,"kb_avail":87846668,"statfs":{"total":96636764160,"available":89954988032,"internally_reserved":0,"allocated":4769951744,"data_stored":5632350256,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":465730,"internal_metadata":1911350462},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,1,9,5,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:14:18 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.61899999999999999,"15min":0.56200000000000006},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":3.3359999999999999,"5min":13.834,"15min":13.834},"last":0.221},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.58799999999999997,"15min":0.54500000000000004},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":1.109,"5min":9.0730000000000004,"15min":9.0730000000000004},"last":0.66400000000000003}]},{"osd":2,"last update":"Tue Oct 17 02:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.54100000000000004,"15min":0.58799999999999997},"min":{"1min":0.20799999999999999,"5min":0.20599999999999999,"15min":0.187},"max":{"1min":3.1709999999999998,"5min":12.382,"15min":12.382},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.308,"5min":0.67800000000000005,"15min":0.61599999999999999},"min":{"1min":0.22600000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.54000000000000004,"5min":12.561,"15min":12.561},"last":0.55600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7184,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4769435648,"data_stored":5632032768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4627996672,"data_stored":5465829395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9399721984,"data_stored":11100151827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:14:29.071 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:14:29.071 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:14:29.687 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:14:29.680+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T02:14:29.688 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:14:29.680+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T02:14:37.547 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:14:37.544+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T02:14:37.563 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:14:37.556+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T02:14:37.590 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:14:37.584+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T02:14:37.591 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:14:37.584+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T02:14:37.591 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:14:37.584+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T02:14:37.591 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:14:37.584+0000 7f31dd33a640 -1 reset not still connected to 0x55e9103c9930 2023-10-17T02:14:37.592 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:14:37.584+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T02:14:49.072 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:14:49.636 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:14:49.637 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:14:49.671 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2893,"stamp":"2023-10-17T02:14:47.672839+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29072681816,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2329221,"num_read_kb":9113953,"num_write":2302343,"num_write_kb":9209023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88447,"ondisk_log_size":88447,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":24027540,"kb_used_data":18418748,"kb_used_omap":1731,"kb_used_meta":5607036,"kb_avail":259087980,"statfs":{"total":289910292480,"available":265306091520,"internally_reserved":0,"allocated":18860797952,"data_stored":22260528867,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1773277,"internal_metadata":5741605155},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,5,8,14,8,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18587648,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4275,"num_read_kb":17048,"num_write":4589,"num_write_kb":18356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002321"},"pg_stats":[{"pgid":"2.7","version":"115'15228","reported_seq":21338,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:44.658559+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:14:44.658559+0000","last_peered":"2023-10-17T02:14:44.658559+0000","last_clean":"2023-10-17T02:14:44.658559+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:14:44.658559+0000","last_undegraded":"2023-10-17T02:14:44.658559+0000","last_fullsized":"2023-10-17T02:14:44.658559+0000","mapping_epoch":13,"log_start":"89'5200","ondisk_log_start":"89'5200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"115'11471","reported_seq":17230,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:44.648811+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:14:44.648811+0000","last_peered":"2023-10-17T02:14:44.648811+0000","last_clean":"2023-10-17T02:14:44.648811+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:14:44.648811+0000","last_undegraded":"2023-10-17T02:14:44.648811+0000","last_fullsized":"2023-10-17T02:14:44.648811+0000","mapping_epoch":13,"log_start":"54'1400","ondisk_log_start":"54'1400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"115'7691","reported_seq":10842,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:44.626074+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:14:44.626074+0000","last_peered":"2023-10-17T02:14:44.626074+0000","last_clean":"2023-10-17T02:14:44.626074+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:14:44.626074+0000","last_undegraded":"2023-10-17T02:14:44.626074+0000","last_fullsized":"2023-10-17T02:14:44.626074+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":7691,"log_dups_size":0,"ondisk_log_size":7691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"115'11507","reported_seq":16676,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:44.652789+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:14:44.652789+0000","last_peered":"2023-10-17T02:14:44.652789+0000","last_clean":"2023-10-17T02:14:44.652789+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:14:44.652789+0000","last_undegraded":"2023-10-17T02:14:44.652789+0000","last_fullsized":"2023-10-17T02:14:44.652789+0000","mapping_epoch":13,"log_start":"56'1500","ondisk_log_start":"56'1500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"115'586403","reported_seq":1169289,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:45.557087+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:14:45.557087+0000","last_peered":"2023-10-17T02:14:45.557087+0000","last_clean":"2023-10-17T02:14:45.557087+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:14:45.557087+0000","last_undegraded":"2023-10-17T02:14:45.557087+0000","last_fullsized":"2023-10-17T02:14:45.557087+0000","mapping_epoch":16,"log_start":"115'576310","ondisk_log_start":"115'576310","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7370223616,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":583562,"num_read_kb":2301044,"num_write":583646,"num_write_kb":2334584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":277,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:12:19.777785+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:12:19.777785+0000","last_peered":"2023-10-17T02:12:19.777785+0000","last_clean":"2023-10-17T02:12:19.777785+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:12:19.777785+0000","last_undegraded":"2023-10-17T02:12:19.777785+0000","last_fullsized":"2023-10-17T02:12:19.777785+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"115'564123","reported_seq":1125733,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:46.419657+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:14:46.419657+0000","last_peered":"2023-10-17T02:14:46.419657+0000","last_clean":"2023-10-17T02:14:46.419657+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:14:46.419657+0000","last_undegraded":"2023-10-17T02:14:46.419657+0000","last_fullsized":"2023-10-17T02:14:46.419657+0000","mapping_epoch":16,"log_start":"115'554119","ondisk_log_start":"115'554119","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":7086784512,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":562283,"num_read_kb":2219220,"num_write":561473,"num_write_kb":2245892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":3114,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:42.948994+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:14:42.948994+0000","last_peered":"2023-10-17T02:14:42.948994+0000","last_clean":"2023-10-17T02:14:42.948994+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:14:42.948994+0000","last_undegraded":"2023-10-17T02:14:42.948994+0000","last_fullsized":"2023-10-17T02:14:42.948994+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"115'587282","reported_seq":1172215,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:45.572968+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:14:45.572968+0000","last_peered":"2023-10-17T02:14:45.572968+0000","last_clean":"2023-10-17T02:14:45.572968+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:14:45.572968+0000","last_undegraded":"2023-10-17T02:14:45.572968+0000","last_fullsized":"2023-10-17T02:14:45.572968+0000","mapping_epoch":16,"log_start":"115'577225","ondisk_log_start":"115'577225","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7386931200,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":585645,"num_read_kb":2310200,"num_write":584528,"num_write_kb":2338112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4664,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:42.949856+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:14:42.949856+0000","last_peered":"2023-10-17T02:14:42.949856+0000","last_clean":"2023-10-17T02:14:42.949856+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:14:42.949856+0000","last_undegraded":"2023-10-17T02:14:42.949856+0000","last_fullsized":"2023-10-17T02:14:42.949856+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"115'574509","reported_seq":1147800,"reported_epoch":115,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:14:46.423659+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:14:46.423659+0000","last_peered":"2023-10-17T02:14:46.423659+0000","last_clean":"2023-10-17T02:14:46.423659+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:14:46.423659+0000","last_undegraded":"2023-10-17T02:14:46.423659+0000","last_fullsized":"2023-10-17T02:14:46.423659+0000","mapping_epoch":16,"log_start":"115'564418","ondisk_log_start":"115'564418","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2127,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 801s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7228149779,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":573769,"num_read_kb":2262836,"num_write":572066,"num_write_kb":2288258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"115'11478","reported_seq":16011,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:44.690947+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:14:44.690947+0000","last_peered":"2023-10-17T02:14:44.690947+0000","last_clean":"2023-10-17T02:14:44.690947+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:14:44.690947+0000","last_undegraded":"2023-10-17T02:14:44.690947+0000","last_fullsized":"2023-10-17T02:14:44.690947+0000","mapping_epoch":13,"log_start":"55'1400","ondisk_log_start":"55'1400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":298,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:12:45.653746+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:12:45.653746+0000","last_peered":"2023-10-17T02:12:45.653746+0000","last_clean":"2023-10-17T02:12:45.653746+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:12:45.653746+0000","last_undegraded":"2023-10-17T02:12:45.653746+0000","last_fullsized":"2023-10-17T02:12:45.653746+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29072089107,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2305259,"num_read_kb":9093300,"num_write":2301713,"num_write_kb":9206846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18857877504,"data_stored":22258737190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40245,"ondisk_log_size":40245,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34223,"internal_metadata":0},"log_size":48103,"ondisk_log_size":48103,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739483,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11127932,"kb_used_data":9210100,"kb_used_omap":852,"kb_used_meta":1916971,"kb_avail":83243908,"statfs":{"total":96636764160,"available":85241761792,"internally_reserved":0,"allocated":9431142400,"data_stored":11130921802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":873467,"internal_metadata":1962978309},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:14:43 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.41099999999999998,"15min":0.45800000000000002},"min":{"1min":0.186,"5min":0.17799999999999999,"15min":0.158},"max":{"1min":0.64600000000000002,"5min":2.3900000000000001,"15min":12.852},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.379,"15min":0.46999999999999997},"min":{"1min":0.20399999999999999,"5min":0.20300000000000001,"15min":0.192},"max":{"1min":0.69199999999999995,"5min":1.9410000000000001,"15min":12.784000000000001},"last":0.66000000000000003}]},{"osd":1,"last update":"Tue Oct 17 02:14:13 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.47299999999999998,"15min":0.44500000000000001},"min":{"1min":0.215,"5min":0.215,"15min":0.20899999999999999},"max":{"1min":1.228,"5min":4.0730000000000004,"15min":6.202},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.495,"15min":0.55200000000000005},"min":{"1min":0.25600000000000001,"5min":0.23899999999999999,"15min":0.23200000000000001},"max":{"1min":1.115,"5min":5.1779999999999999,"15min":12.824999999999999},"last":0.58799999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739479,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6297604,"kb_used_data":4535420,"kb_used_omap":423,"kb_used_meta":1761752,"kb_avail":88074236,"statfs":{"total":96636764160,"available":90188017664,"internally_reserved":0,"allocated":4644270080,"data_stored":5481823081,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":434080,"internal_metadata":1804034144},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,6,6,8,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.64500000000000002,"15min":0.63700000000000001},"min":{"1min":0.24399999999999999,"5min":0.17999999999999999,"15min":0.16600000000000001},"max":{"1min":4.5620000000000003,"5min":5.1109999999999998,"15min":11.891999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.628,"5min":0.68899999999999995,"15min":0.68400000000000005},"min":{"1min":0.27300000000000002,"5min":0.24099999999999999,"15min":0.224},"max":{"1min":4.5979999999999999,"5min":5.1449999999999996,"15min":11.926},"last":0.59399999999999997}]},{"osd":2,"last update":"Tue Oct 17 02:14:37 2023","interfaces":[{"interface":"back","average":{"1min":1.617,"5min":0.76200000000000001,"15min":0.755},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.19500000000000001},"max":{"1min":13.300000000000001,"5min":13.300000000000001,"15min":25.128},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.45000000000000001,"15min":0.49099999999999999},"min":{"1min":0.23000000000000001,"5min":0.219,"15min":0.214},"max":{"1min":3.597,"5min":3.597,"15min":6.7800000000000002},"last":0.38500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739479,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6602004,"kb_used_data":4673228,"kb_used_omap":454,"kb_used_meta":1928313,"kb_avail":87769836,"statfs":{"total":96636764160,"available":89876312064,"internally_reserved":0,"allocated":4785385472,"data_stored":5647783984,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":465730,"internal_metadata":1974592702},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,8,0,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:14:18 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.61899999999999999,"15min":0.56200000000000006},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":3.3359999999999999,"5min":13.834,"15min":13.834},"last":3.105},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.58799999999999997,"15min":0.54500000000000004},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":1.109,"5min":9.0730000000000004,"15min":9.0730000000000004},"last":2.8319999999999999}]},{"osd":2,"last update":"Tue Oct 17 02:14:27 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.437,"15min":0.60599999999999998},"min":{"1min":0.224,"5min":0.20599999999999999,"15min":0.187},"max":{"1min":3.4239999999999999,"5min":3.4239999999999999,"15min":12.382},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.53800000000000003,"15min":0.61099999999999999},"min":{"1min":0.23799999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.55600000000000005,"5min":5.4029999999999996,"15min":12.561},"last":0.36299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7184,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4784840704,"data_stored":5647437824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4642979840,"data_stored":5480812563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9430056960,"data_stored":11130486803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:14:49.673 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:14:49.673 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:15:09.675 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:15:10.183 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:15:10.184 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:15:10.221 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2904,"stamp":"2023-10-17T02:15:09.676701+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29103119192,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2336338,"num_read_kb":9142313,"num_write":2309830,"num_write_kb":9238971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88482,"ondisk_log_size":88482,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":24335900,"kb_used_data":18477952,"kb_used_omap":1731,"kb_used_meta":5856188,"kb_avail":258779620,"statfs":{"total":289910292480,"available":264990330880,"internally_reserved":0,"allocated":18921422848,"data_stored":22321153763,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1773277,"internal_metadata":5996736803},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,6,6,9,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15228928,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3574,"num_read_kb":14248,"num_write":3745,"num_write_kb":14980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002028"},"pg_stats":[{"pgid":"2.7","version":"115'15284","reported_seq":21410,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:15:04.150771+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:15:04.150771+0000","last_peered":"2023-10-17T02:15:04.150771+0000","last_clean":"2023-10-17T02:15:04.150771+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:15:04.150771+0000","last_undegraded":"2023-10-17T02:15:04.150771+0000","last_fullsized":"2023-10-17T02:15:04.150771+0000","mapping_epoch":13,"log_start":"89'5200","ondisk_log_start":"89'5200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"115'11513","reported_seq":17284,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:15:04.146779+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:15:04.146779+0000","last_peered":"2023-10-17T02:15:04.146779+0000","last_clean":"2023-10-17T02:15:04.146779+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:15:04.146779+0000","last_undegraded":"2023-10-17T02:15:04.146779+0000","last_fullsized":"2023-10-17T02:15:04.146779+0000","mapping_epoch":13,"log_start":"56'1500","ondisk_log_start":"56'1500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"115'7719","reported_seq":10878,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:15:04.141445+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:15:04.141445+0000","last_peered":"2023-10-17T02:15:04.141445+0000","last_clean":"2023-10-17T02:15:04.141445+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:15:04.141445+0000","last_undegraded":"2023-10-17T02:15:04.141445+0000","last_fullsized":"2023-10-17T02:15:04.141445+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":7719,"log_dups_size":0,"ondisk_log_size":7719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"115'11549","reported_seq":16730,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:15:04.150036+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:15:04.150036+0000","last_peered":"2023-10-17T02:15:04.150036+0000","last_clean":"2023-10-17T02:15:04.150036+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:15:04.150036+0000","last_undegraded":"2023-10-17T02:15:04.150036+0000","last_fullsized":"2023-10-17T02:15:04.150036+0000","mapping_epoch":13,"log_start":"56'1500","ondisk_log_start":"56'1500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"115'588299","reported_seq":1172955,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:15:05.654371+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:15:05.654371+0000","last_peered":"2023-10-17T02:15:05.654371+0000","last_clean":"2023-10-17T02:15:05.654371+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:15:05.654371+0000","last_undegraded":"2023-10-17T02:15:05.654371+0000","last_fullsized":"2023-10-17T02:15:05.654371+0000","mapping_epoch":16,"log_start":"115'578210","ondisk_log_start":"115'578210","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7377928192,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":585333,"num_read_kb":2308096,"num_write":585542,"num_write_kb":2342168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":277,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:12:19.777785+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:12:19.777785+0000","last_peered":"2023-10-17T02:12:19.777785+0000","last_clean":"2023-10-17T02:12:19.777785+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:12:19.777785+0000","last_undegraded":"2023-10-17T02:12:19.777785+0000","last_fullsized":"2023-10-17T02:12:19.777785+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"115'565909","reported_seq":1129269,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:15:06.554354+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:15:06.554354+0000","last_peered":"2023-10-17T02:15:06.554354+0000","last_clean":"2023-10-17T02:15:06.554354+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:15:06.554354+0000","last_undegraded":"2023-10-17T02:15:06.554354+0000","last_fullsized":"2023-10-17T02:15:06.554354+0000","mapping_epoch":16,"log_start":"115'555819","ondisk_log_start":"115'555819","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":7094050816,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":564034,"num_read_kb":2226192,"num_write":563259,"num_write_kb":2253036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":3122,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:15:03.425299+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:15:03.425299+0000","last_peered":"2023-10-17T02:15:03.425299+0000","last_clean":"2023-10-17T02:15:03.425299+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:15:03.425299+0000","last_undegraded":"2023-10-17T02:15:03.425299+0000","last_fullsized":"2023-10-17T02:15:03.425299+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"115'589265","reported_seq":1175982,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:15:05.635738+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:15:05.635738+0000","last_peered":"2023-10-17T02:15:05.635738+0000","last_clean":"2023-10-17T02:15:05.635738+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:15:05.635738+0000","last_undegraded":"2023-10-17T02:15:05.635738+0000","last_fullsized":"2023-10-17T02:15:05.635738+0000","mapping_epoch":16,"log_start":"115'579225","ondisk_log_start":"115'579225","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7395000320,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":587430,"num_read_kb":2317320,"num_write":586511,"num_write_kb":2346044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4676,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:15:03.425199+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:15:03.425199+0000","last_peered":"2023-10-17T02:15:03.425199+0000","last_clean":"2023-10-17T02:15:03.425199+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:15:03.425199+0000","last_undegraded":"2023-10-17T02:15:03.425199+0000","last_fullsized":"2023-10-17T02:15:03.425199+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"115'576369","reported_seq":1151483,"reported_epoch":115,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:15:06.601326+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:15:06.601326+0000","last_peered":"2023-10-17T02:15:06.601326+0000","last_clean":"2023-10-17T02:15:06.601326+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:15:06.601326+0000","last_undegraded":"2023-10-17T02:15:06.601326+0000","last_fullsized":"2023-10-17T02:15:06.601326+0000","mapping_epoch":16,"log_start":"115'566318","ondisk_log_start":"115'566318","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2197,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 821s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7235547155,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":575579,"num_read_kb":2270052,"num_write":573888,"num_write_kb":2295546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"115'11520","reported_seq":16065,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:15:04.210372+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:15:04.210372+0000","last_peered":"2023-10-17T02:15:04.210372+0000","last_clean":"2023-10-17T02:15:04.210372+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:15:04.210372+0000","last_undegraded":"2023-10-17T02:15:04.210372+0000","last_fullsized":"2023-10-17T02:15:04.210372+0000","mapping_epoch":13,"log_start":"56'1500","ondisk_log_start":"56'1500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":298,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:12:45.653746+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:12:45.653746+0000","last_peered":"2023-10-17T02:12:45.653746+0000","last_clean":"2023-10-17T02:12:45.653746+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:12:45.653746+0000","last_undegraded":"2023-10-17T02:12:45.653746+0000","last_fullsized":"2023-10-17T02:12:45.653746+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29102526483,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2312376,"num_read_kb":9121660,"num_write":2309200,"num_write_kb":9236794,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18919211008,"data_stored":22320070694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40270,"ondisk_log_size":40270,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34223,"internal_metadata":0},"log_size":48113,"ondisk_log_size":48113,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739487,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11273364,"kb_used_data":9239944,"kb_used_omap":852,"kb_used_meta":2032555,"kb_avail":83098476,"statfs":{"total":96636764160,"available":85092839424,"internally_reserved":0,"allocated":9461702656,"data_stored":11161482058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":873467,"internal_metadata":2081336325},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:14:43 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.41099999999999998,"15min":0.45800000000000002},"min":{"1min":0.186,"5min":0.17799999999999999,"15min":0.158},"max":{"1min":0.64600000000000002,"5min":2.3900000000000001,"15min":12.852},"last":0.248},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.379,"15min":0.46999999999999997},"min":{"1min":0.20399999999999999,"5min":0.20300000000000001,"15min":0.192},"max":{"1min":0.69199999999999995,"5min":1.9410000000000001,"15min":12.784000000000001},"last":0.29399999999999998}]},{"osd":1,"last update":"Tue Oct 17 02:14:13 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.47299999999999998,"15min":0.44500000000000001},"min":{"1min":0.215,"5min":0.215,"15min":0.20899999999999999},"max":{"1min":1.228,"5min":4.0730000000000004,"15min":6.202},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.495,"15min":0.55200000000000005},"min":{"1min":0.25600000000000001,"5min":0.23899999999999999,"15min":0.23200000000000001},"max":{"1min":1.115,"5min":5.1779999999999999,"15min":12.824999999999999},"last":0.371}]}]},{"osd":1,"up_from":8,"seq":34359739483,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6390940,"kb_used_data":4549332,"kb_used_omap":423,"kb_used_meta":1841176,"kb_avail":87980900,"statfs":{"total":96636764160,"available":90092441600,"internally_reserved":0,"allocated":4658515968,"data_stored":5496068969,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":434080,"internal_metadata":1885364320},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,3,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.64500000000000002,"15min":0.63700000000000001},"min":{"1min":0.24399999999999999,"5min":0.17999999999999999,"15min":0.16600000000000001},"max":{"1min":4.5620000000000003,"5min":5.1109999999999998,"15min":11.891999999999999},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.68899999999999995,"15min":0.68400000000000005},"min":{"1min":0.27300000000000002,"5min":0.24099999999999999,"15min":0.224},"max":{"1min":4.5979999999999999,"5min":5.1449999999999996,"15min":11.926},"last":0.309}]},{"osd":2,"last update":"Tue Oct 17 02:14:37 2023","interfaces":[{"interface":"back","average":{"1min":1.617,"5min":0.76200000000000001,"15min":0.755},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.19500000000000001},"max":{"1min":13.300000000000001,"5min":13.300000000000001,"15min":25.128},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.45000000000000001,"15min":0.49099999999999999},"min":{"1min":0.23000000000000001,"5min":0.219,"15min":0.214},"max":{"1min":3.597,"5min":3.597,"15min":6.7800000000000002},"last":0.33300000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739483,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6671596,"kb_used_data":4688676,"kb_used_omap":454,"kb_used_meta":1982457,"kb_avail":87700244,"statfs":{"total":96636764160,"available":89805049856,"internally_reserved":0,"allocated":4801204224,"data_stored":5663602736,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":465730,"internal_metadata":2030036158},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,2,6,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:14:18 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.61899999999999999,"15min":0.56200000000000006},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":3.3359999999999999,"5min":13.834,"15min":13.834},"last":0.221},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.58799999999999997,"15min":0.54500000000000004},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":1.109,"5min":9.0730000000000004,"15min":9.0730000000000004},"last":0.247}]},{"osd":2,"last update":"Tue Oct 17 02:14:27 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.437,"15min":0.60599999999999998},"min":{"1min":0.224,"5min":0.20599999999999999,"15min":0.187},"max":{"1min":3.4239999999999999,"5min":3.4239999999999999,"15min":12.382},"last":2.9689999999999999},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.53800000000000003,"15min":0.61099999999999999},"min":{"1min":0.23799999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.55600000000000005,"5min":5.4029999999999996,"15min":12.561},"last":2.706}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7184,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4800729088,"data_stored":5663326208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4657786880,"data_stored":5495619603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9460695040,"data_stored":11161124883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:15:10.223 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:15:10.224 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:15:30.225 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:15:30.736 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:15:30.737 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:15:30.782 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2914,"stamp":"2023-10-17T02:15:29.680250+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29134199640,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2343773,"num_read_kb":9171953,"num_write":2317450,"num_write_kb":9269451,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88485,"ondisk_log_size":88485,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23936708,"kb_used_data":18539560,"kb_used_omap":1731,"kb_used_meta":5395388,"kb_avail":259178812,"statfs":{"total":289910292480,"available":265399103488,"internally_reserved":0,"allocated":18984509440,"data_stored":22384240355,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1773277,"internal_metadata":5524877603},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,19,6,11,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15699968,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3809,"num_read_kb":15188,"num_write":3849,"num_write_kb":15396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002244"},"pg_stats":[{"pgid":"2.7","version":"115'15300","reported_seq":21442,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:15:23.908302+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:15:23.908302+0000","last_peered":"2023-10-17T02:15:23.908302+0000","last_clean":"2023-10-17T02:15:23.908302+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:15:23.908302+0000","last_undegraded":"2023-10-17T02:15:23.908302+0000","last_fullsized":"2023-10-17T02:15:23.908302+0000","mapping_epoch":13,"log_start":"89'5200","ondisk_log_start":"89'5200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"115'11525","reported_seq":17308,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:15:23.907444+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:15:23.907444+0000","last_peered":"2023-10-17T02:15:23.907444+0000","last_clean":"2023-10-17T02:15:23.907444+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:15:23.907444+0000","last_undegraded":"2023-10-17T02:15:23.907444+0000","last_fullsized":"2023-10-17T02:15:23.907444+0000","mapping_epoch":13,"log_start":"56'1500","ondisk_log_start":"56'1500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"115'7727","reported_seq":10894,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:15:23.907570+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:15:23.907570+0000","last_peered":"2023-10-17T02:15:23.907570+0000","last_clean":"2023-10-17T02:15:23.907570+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:15:23.907570+0000","last_undegraded":"2023-10-17T02:15:23.907570+0000","last_fullsized":"2023-10-17T02:15:23.907570+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":7727,"log_dups_size":0,"ondisk_log_size":7727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"115'11561","reported_seq":16754,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:15:23.907433+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:15:23.907433+0000","last_peered":"2023-10-17T02:15:23.907433+0000","last_clean":"2023-10-17T02:15:23.907433+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:15:23.907433+0000","last_undegraded":"2023-10-17T02:15:23.907433+0000","last_fullsized":"2023-10-17T02:15:23.907433+0000","mapping_epoch":13,"log_start":"56'1500","ondisk_log_start":"56'1500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"115'590267","reported_seq":1176859,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:15:25.720517+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:15:25.720517+0000","last_peered":"2023-10-17T02:15:25.720517+0000","last_clean":"2023-10-17T02:15:25.720517+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:15:25.720517+0000","last_undegraded":"2023-10-17T02:15:25.720517+0000","last_fullsized":"2023-10-17T02:15:25.720517+0000","mapping_epoch":16,"log_start":"115'580210","ondisk_log_start":"115'580210","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7385980928,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":587270,"num_read_kb":2315832,"num_write":587510,"num_write_kb":2350040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":277,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:12:19.777785+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:12:19.777785+0000","last_peered":"2023-10-17T02:12:19.777785+0000","last_clean":"2023-10-17T02:12:19.777785+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:12:19.777785+0000","last_undegraded":"2023-10-17T02:12:19.777785+0000","last_fullsized":"2023-10-17T02:12:19.777785+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"115'567725","reported_seq":1132871,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:15:26.742999+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:15:26.742999+0000","last_peered":"2023-10-17T02:15:26.742999+0000","last_clean":"2023-10-17T02:15:26.742999+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:15:26.742999+0000","last_undegraded":"2023-10-17T02:15:26.742999+0000","last_fullsized":"2023-10-17T02:15:26.742999+0000","mapping_epoch":16,"log_start":"115'557719","ondisk_log_start":"115'557719","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":7101435904,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":565820,"num_read_kb":2233320,"num_write":565075,"num_write_kb":2260300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":3130,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:15:23.921754+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:15:23.921754+0000","last_peered":"2023-10-17T02:15:23.921754+0000","last_clean":"2023-10-17T02:15:23.921754+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:15:23.921754+0000","last_undegraded":"2023-10-17T02:15:23.921754+0000","last_fullsized":"2023-10-17T02:15:23.921754+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"115'591203","reported_seq":1179770,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:15:25.728308+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:15:25.728308+0000","last_peered":"2023-10-17T02:15:25.728308+0000","last_clean":"2023-10-17T02:15:25.728308+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:15:25.728308+0000","last_undegraded":"2023-10-17T02:15:25.728308+0000","last_fullsized":"2023-10-17T02:15:25.728308+0000","mapping_epoch":16,"log_start":"115'581125","ondisk_log_start":"115'581125","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7402885120,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":589275,"num_read_kb":2324660,"num_write":588449,"num_write_kb":2353796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4688,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:15:23.908108+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:15:23.908108+0000","last_peered":"2023-10-17T02:15:23.908108+0000","last_clean":"2023-10-17T02:15:23.908108+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:15:23.908108+0000","last_undegraded":"2023-10-17T02:15:23.908108+0000","last_fullsized":"2023-10-17T02:15:23.908108+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"115'578290","reported_seq":1155275,"reported_epoch":115,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:15:26.746743+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:15:26.746743+0000","last_peered":"2023-10-17T02:15:26.746743+0000","last_clean":"2023-10-17T02:15:26.746743+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:15:26.746743+0000","last_undegraded":"2023-10-17T02:15:26.746743+0000","last_fullsized":"2023-10-17T02:15:26.746743+0000","mapping_epoch":16,"log_start":"115'568218","ondisk_log_start":"115'568218","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2233,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 841s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7243304979,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":577446,"num_read_kb":2277488,"num_write":575786,"num_write_kb":2303138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"115'11532","reported_seq":16089,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:15:23.907963+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:15:23.907963+0000","last_peered":"2023-10-17T02:15:23.907963+0000","last_clean":"2023-10-17T02:15:23.907963+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:15:23.907963+0000","last_undegraded":"2023-10-17T02:15:23.907963+0000","last_fullsized":"2023-10-17T02:15:23.907963+0000","mapping_epoch":13,"log_start":"56'1500","ondisk_log_start":"56'1500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":298,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:12:45.653746+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:12:45.653746+0000","last_peered":"2023-10-17T02:12:45.653746+0000","last_clean":"2023-10-17T02:12:45.653746+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:12:45.653746+0000","last_undegraded":"2023-10-17T02:12:45.653746+0000","last_fullsized":"2023-10-17T02:12:45.653746+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29133606931,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2319811,"num_read_kb":9151300,"num_write":2316820,"num_write_kb":9267274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18981695488,"data_stored":22382555174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40213,"ondisk_log_size":40213,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34223,"internal_metadata":0},"log_size":48173,"ondisk_log_size":48173,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739491,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11442680,"kb_used_data":9270448,"kb_used_omap":852,"kb_used_meta":2171371,"kb_avail":82929160,"statfs":{"total":96636764160,"available":84919459840,"internally_reserved":0,"allocated":9492938752,"data_stored":11192718154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":873467,"internal_metadata":2223483909},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:14:43 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.41099999999999998,"15min":0.45800000000000002},"min":{"1min":0.186,"5min":0.17799999999999999,"15min":0.158},"max":{"1min":0.64600000000000002,"5min":2.3900000000000001,"15min":12.852},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.379,"15min":0.46999999999999997},"min":{"1min":0.20399999999999999,"5min":0.20300000000000001,"15min":0.192},"max":{"1min":0.69199999999999995,"5min":1.9410000000000001,"15min":12.784000000000001},"last":0.23499999999999999}]},{"osd":1,"last update":"Tue Oct 17 02:15:13 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.45400000000000001,"15min":0.45700000000000002},"min":{"1min":0.247,"5min":0.215,"15min":0.20899999999999999},"max":{"1min":3.302,"5min":3.302,"15min":6.202},"last":0.25},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.46400000000000002,"15min":0.55300000000000005},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.59399999999999997,"5min":2.819,"15min":12.824999999999999},"last":0.28499999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739487,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6437640,"kb_used_data":4564096,"kb_used_omap":423,"kb_used_meta":1873112,"kb_avail":87934200,"statfs":{"total":96636764160,"available":90044620800,"internally_reserved":0,"allocated":4673634304,"data_stored":5511187305,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":434080,"internal_metadata":1918066784},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,8,3,5,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.64500000000000002,"15min":0.63700000000000001},"min":{"1min":0.24399999999999999,"5min":0.17999999999999999,"15min":0.16600000000000001},"max":{"1min":4.5620000000000003,"5min":5.1109999999999998,"15min":11.891999999999999},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.68899999999999995,"15min":0.68400000000000005},"min":{"1min":0.27300000000000002,"5min":0.24099999999999999,"15min":0.224},"max":{"1min":4.5979999999999999,"5min":5.1449999999999996,"15min":11.926},"last":0.42399999999999999}]},{"osd":2,"last update":"Tue Oct 17 02:14:37 2023","interfaces":[{"interface":"back","average":{"1min":1.617,"5min":0.76200000000000001,"15min":0.755},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.19500000000000001},"max":{"1min":13.300000000000001,"5min":13.300000000000001,"15min":25.128},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.45000000000000001,"15min":0.49099999999999999},"min":{"1min":0.23000000000000001,"5min":0.219,"15min":0.214},"max":{"1min":3.597,"5min":3.597,"15min":6.7800000000000002},"last":0.26600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739487,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6056388,"kb_used_data":4705016,"kb_used_omap":454,"kb_used_meta":1350905,"kb_avail":88315452,"statfs":{"total":96636764160,"available":90435022848,"internally_reserved":0,"allocated":4817936384,"data_stored":5680334896,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":465730,"internal_metadata":1383326910},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,8,2,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:15:21 2023","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":0.77400000000000002,"15min":0.61499999999999999},"min":{"1min":0.184,"5min":0.161,"15min":0.161},"max":{"1min":6.0389999999999997,"5min":13.834,"15min":13.834},"last":0.184},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.52000000000000002,"15min":0.55400000000000005},"min":{"1min":0.215,"5min":0.185,"15min":0.185},"max":{"1min":2.8319999999999999,"5min":9.0730000000000004,"15min":9.0730000000000004},"last":0.252}]},{"osd":2,"last update":"Tue Oct 17 02:14:27 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.437,"15min":0.60599999999999998},"min":{"1min":0.224,"5min":0.20599999999999999,"15min":0.187},"max":{"1min":3.4239999999999999,"5min":3.4239999999999999,"15min":12.382},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.53800000000000003,"15min":0.61099999999999999},"min":{"1min":0.23799999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.55600000000000005,"5min":5.4029999999999996,"15min":12.561},"last":0.221}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7184,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4816740352,"data_stored":5679337472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4672962560,"data_stored":5510795283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9491992576,"data_stored":11192422419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:15:30.784 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:15:30.785 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-17T02:15:31.360 INFO:teuthology.orchestra.run.smithi169.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-17T02:15:31.391 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:15:32.128 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:15:32.123+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T02:15:32.139 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:15:32.131+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T02:15:51.392 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:15:52.000 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:15:52.002 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:15:52.041 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2925,"stamp":"2023-10-17T02:15:51.684070+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29168642904,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2352102,"num_read_kb":9205189,"num_write":2325893,"num_write_kb":9303223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88530,"ondisk_log_size":88530,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23556524,"kb_used_data":18602836,"kb_used_omap":1733,"kb_used_meta":4951930,"kb_avail":259558996,"statfs":{"total":289910292480,"available":265788411904,"internally_reserved":0,"allocated":19049304064,"data_stored":22449034979,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1774957,"internal_metadata":5070776979},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,10,3,9,11,8,11],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19546112,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4714,"num_read_kb":18820,"num_write":4794,"num_write_kb":19176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002093"},"pg_stats":[{"pgid":"2.7","version":"115'15349","reported_seq":21507,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:15:46.864490+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:15:46.864490+0000","last_peered":"2023-10-17T02:15:46.864490+0000","last_clean":"2023-10-17T02:15:46.864490+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:15:46.864490+0000","last_undegraded":"2023-10-17T02:15:46.864490+0000","last_fullsized":"2023-10-17T02:15:46.864490+0000","mapping_epoch":13,"log_start":"89'5300","ondisk_log_start":"89'5300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"115'11562","reported_seq":17358,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:15:46.842875+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:15:46.842875+0000","last_peered":"2023-10-17T02:15:46.842875+0000","last_clean":"2023-10-17T02:15:46.842875+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:15:46.842875+0000","last_undegraded":"2023-10-17T02:15:46.842875+0000","last_fullsized":"2023-10-17T02:15:46.842875+0000","mapping_epoch":13,"log_start":"56'1500","ondisk_log_start":"56'1500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"115'7752","reported_seq":10927,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:15:46.824276+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:15:46.824276+0000","last_peered":"2023-10-17T02:15:46.824276+0000","last_clean":"2023-10-17T02:15:46.824276+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:15:46.824276+0000","last_undegraded":"2023-10-17T02:15:46.824276+0000","last_fullsized":"2023-10-17T02:15:46.824276+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":7752,"log_dups_size":0,"ondisk_log_size":7752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"115'11598","reported_seq":16803,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:15:46.851598+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:15:46.851598+0000","last_peered":"2023-10-17T02:15:46.851598+0000","last_clean":"2023-10-17T02:15:46.851598+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:15:46.851598+0000","last_undegraded":"2023-10-17T02:15:46.851598+0000","last_fullsized":"2023-10-17T02:15:46.851598+0000","mapping_epoch":13,"log_start":"56'1500","ondisk_log_start":"56'1500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"115'592679","reported_seq":1181566,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:15:50.904174+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:15:50.904174+0000","last_peered":"2023-10-17T02:15:50.904174+0000","last_clean":"2023-10-17T02:15:50.904174+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:15:50.904174+0000","last_undegraded":"2023-10-17T02:15:50.904174+0000","last_fullsized":"2023-10-17T02:15:50.904174+0000","mapping_epoch":16,"log_start":"115'582610","ondisk_log_start":"115'582610","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7395774464,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":589562,"num_read_kb":2324984,"num_write":589922,"num_write_kb":2359688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":277,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:12:19.777785+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:12:19.777785+0000","last_peered":"2023-10-17T02:12:19.777785+0000","last_clean":"2023-10-17T02:12:19.777785+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:12:19.777785+0000","last_undegraded":"2023-10-17T02:12:19.777785+0000","last_fullsized":"2023-10-17T02:12:19.777785+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"115'569579","reported_seq":1136547,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:15:46.854027+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:15:46.854027+0000","last_peered":"2023-10-17T02:15:46.854027+0000","last_clean":"2023-10-17T02:15:46.854027+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:15:46.854027+0000","last_undegraded":"2023-10-17T02:15:46.854027+0000","last_fullsized":"2023-10-17T02:15:46.854027+0000","mapping_epoch":16,"log_start":"115'559519","ondisk_log_start":"115'559519","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":7108997120,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":567642,"num_read_kb":2240584,"num_write":566929,"num_write_kb":2267716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":3138,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:15:44.392315+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:15:44.392315+0000","last_peered":"2023-10-17T02:15:44.392315+0000","last_clean":"2023-10-17T02:15:44.392315+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:15:44.392315+0000","last_undegraded":"2023-10-17T02:15:44.392315+0000","last_fullsized":"2023-10-17T02:15:44.392315+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"115'593550","reported_seq":1184402,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:15:50.889069+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:15:50.889069+0000","last_peered":"2023-10-17T02:15:50.889069+0000","last_clean":"2023-10-17T02:15:50.889069+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:15:50.889069+0000","last_undegraded":"2023-10-17T02:15:50.889069+0000","last_fullsized":"2023-10-17T02:15:50.889069+0000","mapping_epoch":16,"log_start":"115'583525","ondisk_log_start":"115'583525","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7412494336,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":591562,"num_read_kb":2333784,"num_write":590796,"num_write_kb":2363184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4700,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:15:44.391041+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:15:44.391041+0000","last_peered":"2023-10-17T02:15:44.391041+0000","last_clean":"2023-10-17T02:15:44.391041+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:15:44.391041+0000","last_undegraded":"2023-10-17T02:15:44.391041+0000","last_fullsized":"2023-10-17T02:15:44.391041+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"115'580137","reported_seq":1159062,"reported_epoch":115,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:15:46.864478+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:15:46.864478+0000","last_peered":"2023-10-17T02:15:46.864478+0000","last_clean":"2023-10-17T02:15:46.864478+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:15:46.864478+0000","last_undegraded":"2023-10-17T02:15:46.864478+0000","last_fullsized":"2023-10-17T02:15:46.864478+0000","mapping_epoch":16,"log_start":"115'570118","ondisk_log_start":"115'570118","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2267,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 861s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7250784275,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":579374,"num_read_kb":2285184,"num_write":577616,"num_write_kb":2310458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"115'11569","reported_seq":16136,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:15:44.387661+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:15:44.387661+0000","last_peered":"2023-10-17T02:15:44.387661+0000","last_clean":"2023-10-17T02:15:44.387661+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:15:44.387661+0000","last_undegraded":"2023-10-17T02:15:44.387661+0000","last_fullsized":"2023-10-17T02:15:44.387661+0000","mapping_epoch":13,"log_start":"56'1500","ondisk_log_start":"56'1500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":298,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:12:45.653746+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:12:45.653746+0000","last_peered":"2023-10-17T02:12:45.653746+0000","last_clean":"2023-10-17T02:12:45.653746+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:12:45.653746+0000","last_undegraded":"2023-10-17T02:12:45.653746+0000","last_fullsized":"2023-10-17T02:12:45.653746+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29168050195,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2328140,"num_read_kb":9184536,"num_write":2325263,"num_write_kb":9301046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19046674432,"data_stored":22447534118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40173,"ondisk_log_size":40173,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34223,"internal_metadata":0},"log_size":48258,"ondisk_log_size":48258,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739495,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10908812,"kb_used_data":9300036,"kb_used_omap":854,"kb_used_meta":1607913,"kb_avail":83463028,"statfs":{"total":96636764160,"available":85466140672,"internally_reserved":0,"allocated":9523236864,"data_stored":11223016266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":875147,"internal_metadata":1646503285},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,2,8,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:15:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.42999999999999999,"15min":0.47299999999999998},"min":{"1min":0.187,"5min":0.17799999999999999,"15min":0.158},"max":{"1min":4.4450000000000003,"5min":4.4450000000000003,"15min":12.852},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.40300000000000002,"15min":0.47499999999999998},"min":{"1min":0.22,"5min":0.20300000000000001,"15min":0.192},"max":{"1min":3.2109999999999999,"5min":3.2109999999999999,"15min":12.784000000000001},"last":0.29799999999999999}]},{"osd":1,"last update":"Tue Oct 17 02:15:13 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.45400000000000001,"15min":0.45700000000000002},"min":{"1min":0.247,"5min":0.215,"15min":0.20899999999999999},"max":{"1min":3.302,"5min":3.302,"15min":6.202},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.46400000000000002,"15min":0.55300000000000005},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.59399999999999997,"5min":2.819,"15min":12.824999999999999},"last":0.70199999999999996}]}]},{"osd":1,"up_from":8,"seq":34359739491,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6514052,"kb_used_data":4578812,"kb_used_omap":423,"kb_used_meta":1934808,"kb_avail":87857788,"statfs":{"total":96636764160,"available":89966374912,"internally_reserved":0,"allocated":4688703488,"data_stored":5526256489,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":434080,"internal_metadata":1981243488},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,1,0,5,8,11],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:15:41 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.57599999999999996,"15min":0.63400000000000001},"min":{"1min":0.20200000000000001,"5min":0.17999999999999999,"15min":0.16600000000000001},"max":{"1min":0.61899999999999999,"5min":5.1109999999999998,"15min":11.891999999999999},"last":0.316},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.623,"15min":0.68100000000000005},"min":{"1min":0.30499999999999999,"5min":0.24099999999999999,"15min":0.224},"max":{"1min":0.64600000000000002,"5min":5.1449999999999996,"15min":11.926},"last":0.48899999999999999}]},{"osd":2,"last update":"Tue Oct 17 02:14:37 2023","interfaces":[{"interface":"back","average":{"1min":1.617,"5min":0.76200000000000001,"15min":0.755},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.19500000000000001},"max":{"1min":13.300000000000001,"5min":13.300000000000001,"15min":25.128},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.45000000000000001,"15min":0.49099999999999999},"min":{"1min":0.23000000000000001,"5min":0.219,"15min":0.214},"max":{"1min":3.597,"5min":3.597,"15min":6.7800000000000002},"last":0.32000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739492,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6133660,"kb_used_data":4723988,"kb_used_omap":454,"kb_used_meta":1409209,"kb_avail":88238180,"statfs":{"total":96636764160,"available":90355896320,"internally_reserved":0,"allocated":4837363712,"data_stored":5699762224,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":465730,"internal_metadata":1443030206},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,0,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:15:21 2023","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":0.77400000000000002,"15min":0.61499999999999999},"min":{"1min":0.184,"5min":0.161,"15min":0.161},"max":{"1min":6.0389999999999997,"5min":13.834,"15min":13.834},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.52000000000000002,"15min":0.55400000000000005},"min":{"1min":0.215,"5min":0.185,"15min":0.185},"max":{"1min":2.8319999999999999,"5min":9.0730000000000004,"15min":9.0730000000000004},"last":0.442}]},{"osd":2,"last update":"Tue Oct 17 02:15:29 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.48899999999999999,"15min":0.60899999999999999},"min":{"1min":0.23100000000000001,"5min":0.20599999999999999,"15min":0.187},"max":{"1min":2.9689999999999999,"5min":3.4239999999999999,"15min":12.382},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.44600000000000001,"15min":0.60799999999999998},"min":{"1min":0.221,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":2.706,"5min":3.8780000000000001,"15min":12.561},"last":0.47899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7184,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4836225024,"data_stored":5698822144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4688052224,"data_stored":5525884947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9522397184,"data_stored":11222827027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:15:52.043 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:15:52.043 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:15:52.937 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:15:52.934+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T02:15:52.940 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:15:52.938+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fc7af70 2023-10-17T02:15:52.940 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:15:52.938+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T02:15:52.941 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:15:52.938+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea5ee0 2023-10-17T02:15:52.941 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:15:52.938+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T02:15:52.956 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:15:52.946+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b5f0 2023-10-17T02:15:52.956 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:15:52.946+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T02:15:52.956 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:15:52.950+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T02:15:52.956 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:15:52.950+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T02:16:04.873 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:16:04.870+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T02:16:04.891 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:16:04.882+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T02:16:04.891 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:16:04.882+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T02:16:04.907 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:16:04.902+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T02:16:12.045 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:16:12.553 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:16:12.554 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:16:12.582 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2935,"stamp":"2023-10-17T02:16:11.689694+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29198498648,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2359528,"num_read_kb":9234865,"num_write":2333257,"num_write_kb":9332679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88608,"ondisk_log_size":88608,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23772472,"kb_used_data":18660448,"kb_used_omap":1733,"kb_used_meta":5110266,"kb_avail":259343048,"statfs":{"total":289910292480,"available":265567281152,"internally_reserved":0,"allocated":19108298752,"data_stored":22508029667,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1774957,"internal_metadata":5232913043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,2,5,14,18,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18661376,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4619,"num_read_kb":18468,"num_write":4611,"num_write_kb":18444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003424"},"pg_stats":[{"pgid":"2.7","version":"115'15396","reported_seq":21571,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:16:04.910519+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:16:04.910519+0000","last_peered":"2023-10-17T02:16:04.910519+0000","last_clean":"2023-10-17T02:16:04.910519+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:16:04.910519+0000","last_undegraded":"2023-10-17T02:16:04.910519+0000","last_fullsized":"2023-10-17T02:16:04.910519+0000","mapping_epoch":13,"log_start":"89'5300","ondisk_log_start":"89'5300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"115'11598","reported_seq":17406,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:16:04.898734+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:16:04.898734+0000","last_peered":"2023-10-17T02:16:04.898734+0000","last_clean":"2023-10-17T02:16:04.898734+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:16:04.898734+0000","last_undegraded":"2023-10-17T02:16:04.898734+0000","last_fullsized":"2023-10-17T02:16:04.898734+0000","mapping_epoch":13,"log_start":"56'1500","ondisk_log_start":"56'1500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"115'7776","reported_seq":10959,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:16:04.875041+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:16:04.875041+0000","last_peered":"2023-10-17T02:16:04.875041+0000","last_clean":"2023-10-17T02:16:04.875041+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:16:04.875041+0000","last_undegraded":"2023-10-17T02:16:04.875041+0000","last_fullsized":"2023-10-17T02:16:04.875041+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":7776,"log_dups_size":0,"ondisk_log_size":7776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"115'11635","reported_seq":16852,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:16:04.893895+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:16:04.893895+0000","last_peered":"2023-10-17T02:16:04.893895+0000","last_clean":"2023-10-17T02:16:04.893895+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:16:04.893895+0000","last_undegraded":"2023-10-17T02:16:04.893895+0000","last_fullsized":"2023-10-17T02:16:04.893895+0000","mapping_epoch":13,"log_start":"58'1600","ondisk_log_start":"58'1600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"115'594601","reported_seq":1185323,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:16:11.013493+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:16:11.013493+0000","last_peered":"2023-10-17T02:16:11.013493+0000","last_clean":"2023-10-17T02:16:11.013493+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:16:11.013493+0000","last_undegraded":"2023-10-17T02:16:11.013493+0000","last_fullsized":"2023-10-17T02:16:11.013493+0000","mapping_epoch":16,"log_start":"115'584510","ondisk_log_start":"115'584510","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7403569152,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":591406,"num_read_kb":2332352,"num_write":591844,"num_write_kb":2367376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":277,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:12:19.777785+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:12:19.777785+0000","last_peered":"2023-10-17T02:12:19.777785+0000","last_clean":"2023-10-17T02:12:19.777785+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:12:19.777785+0000","last_undegraded":"2023-10-17T02:12:19.777785+0000","last_fullsized":"2023-10-17T02:12:19.777785+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"115'571288","reported_seq":1140035,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:16:06.920240+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:16:06.920240+0000","last_peered":"2023-10-17T02:16:06.920240+0000","last_clean":"2023-10-17T02:16:06.920240+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:16:06.920240+0000","last_undegraded":"2023-10-17T02:16:06.920240+0000","last_fullsized":"2023-10-17T02:16:06.920240+0000","mapping_epoch":16,"log_start":"115'561219","ondisk_log_start":"115'561219","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":7115898880,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":569423,"num_read_kb":2247708,"num_write":568638,"num_write_kb":2274552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":3146,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:16:04.885677+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:16:04.885677+0000","last_peered":"2023-10-17T02:16:04.885677+0000","last_clean":"2023-10-17T02:16:04.885677+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:16:04.885677+0000","last_undegraded":"2023-10-17T02:16:04.885677+0000","last_fullsized":"2023-10-17T02:16:04.885677+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"115'595468","reported_seq":1188259,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:16:11.010830+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:16:11.010830+0000","last_peered":"2023-10-17T02:16:11.010830+0000","last_clean":"2023-10-17T02:16:11.010830+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:16:11.010830+0000","last_undegraded":"2023-10-17T02:16:11.010830+0000","last_fullsized":"2023-10-17T02:16:11.010830+0000","mapping_epoch":16,"log_start":"115'585425","ondisk_log_start":"115'585425","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7420268544,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":593499,"num_read_kb":2341528,"num_write":592714,"num_write_kb":2370856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4712,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:16:04.916633+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:16:04.916633+0000","last_peered":"2023-10-17T02:16:04.916633+0000","last_clean":"2023-10-17T02:16:04.916633+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:16:04.916633+0000","last_undegraded":"2023-10-17T02:16:04.916633+0000","last_fullsized":"2023-10-17T02:16:04.916633+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"115'581985","reported_seq":1162778,"reported_epoch":115,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:16:06.929741+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:16:06.929741+0000","last_peered":"2023-10-17T02:16:06.929741+0000","last_clean":"2023-10-17T02:16:06.929741+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:16:06.929741+0000","last_undegraded":"2023-10-17T02:16:06.929741+0000","last_fullsized":"2023-10-17T02:16:06.929741+0000","mapping_epoch":16,"log_start":"115'571918","ondisk_log_start":"115'571918","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2323,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 881s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7258169363,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":581238,"num_read_kb":2292624,"num_write":579431,"num_write_kb":2317718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"115'11606","reported_seq":16187,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:16:04.937123+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:16:04.937123+0000","last_peered":"2023-10-17T02:16:04.937123+0000","last_clean":"2023-10-17T02:16:04.937123+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:16:04.937123+0000","last_undegraded":"2023-10-17T02:16:04.937123+0000","last_fullsized":"2023-10-17T02:16:04.937123+0000","mapping_epoch":13,"log_start":"58'1600","ondisk_log_start":"58'1600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":298,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:12:45.653746+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:12:45.653746+0000","last_peered":"2023-10-17T02:12:45.653746+0000","last_clean":"2023-10-17T02:12:45.653746+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:12:45.653746+0000","last_undegraded":"2023-10-17T02:12:45.653746+0000","last_fullsized":"2023-10-17T02:12:45.653746+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29197905939,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2335566,"num_read_kb":9214212,"num_write":2332627,"num_write_kb":9330502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19107147776,"data_stored":22508007462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40270,"ondisk_log_size":40270,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34223,"internal_metadata":0},"log_size":48239,"ondisk_log_size":48239,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739499,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10957172,"kb_used_data":9328876,"kb_used_omap":854,"kb_used_meta":1627433,"kb_avail":83414668,"statfs":{"total":96636764160,"available":85416620032,"internally_reserved":0,"allocated":9552769024,"data_stored":11252548426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":875147,"internal_metadata":1666491765},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:15:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.42999999999999999,"15min":0.47299999999999998},"min":{"1min":0.187,"5min":0.17799999999999999,"15min":0.158},"max":{"1min":4.4450000000000003,"5min":4.4450000000000003,"15min":12.852},"last":0.183},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.40300000000000002,"15min":0.47499999999999998},"min":{"1min":0.22,"5min":0.20300000000000001,"15min":0.192},"max":{"1min":3.2109999999999999,"5min":3.2109999999999999,"15min":12.784000000000001},"last":0.24399999999999999}]},{"osd":1,"last update":"Tue Oct 17 02:15:13 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.45400000000000001,"15min":0.45700000000000002},"min":{"1min":0.247,"5min":0.215,"15min":0.20899999999999999},"max":{"1min":3.302,"5min":3.302,"15min":6.202},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.46400000000000002,"15min":0.55300000000000005},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.59399999999999997,"5min":2.819,"15min":12.824999999999999},"last":0.432}]}]},{"osd":1,"up_from":8,"seq":34359739495,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6616952,"kb_used_data":4592880,"kb_used_omap":423,"kb_used_meta":2023640,"kb_avail":87754888,"statfs":{"total":96636764160,"available":89861005312,"internally_reserved":0,"allocated":4703109120,"data_stored":5540662121,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":434080,"internal_metadata":2072207456},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,4,11,13,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:15:41 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.57599999999999996,"15min":0.63400000000000001},"min":{"1min":0.20200000000000001,"5min":0.17999999999999999,"15min":0.16600000000000001},"max":{"1min":0.61899999999999999,"5min":5.1109999999999998,"15min":11.891999999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.623,"15min":0.68100000000000005},"min":{"1min":0.30499999999999999,"5min":0.24099999999999999,"15min":0.224},"max":{"1min":0.64600000000000002,"5min":5.1449999999999996,"15min":11.926},"last":0.32200000000000001}]},{"osd":2,"last update":"Tue Oct 17 02:15:47 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.745,"15min":0.751},"min":{"1min":0.23200000000000001,"5min":0.20599999999999999,"15min":0.19500000000000001},"max":{"1min":0.54600000000000004,"5min":13.300000000000001,"15min":25.128},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.45200000000000001,"15min":0.48699999999999999},"min":{"1min":0.245,"5min":0.219,"15min":0.214},"max":{"1min":0.57899999999999996,"5min":3.597,"15min":6.7800000000000002},"last":0.30299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739496,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6198348,"kb_used_data":4738692,"kb_used_omap":454,"kb_used_meta":1459193,"kb_avail":88173492,"statfs":{"total":96636764160,"available":90289655808,"internally_reserved":0,"allocated":4852420608,"data_stored":5714819120,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":465730,"internal_metadata":1494213822},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:15:21 2023","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":0.77400000000000002,"15min":0.61499999999999999},"min":{"1min":0.184,"5min":0.161,"15min":0.161},"max":{"1min":6.0389999999999997,"5min":13.834,"15min":13.834},"last":0.19700000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.52000000000000002,"15min":0.55400000000000005},"min":{"1min":0.215,"5min":0.185,"15min":0.185},"max":{"1min":2.8319999999999999,"5min":9.0730000000000004,"15min":9.0730000000000004},"last":0.248}]},{"osd":2,"last update":"Tue Oct 17 02:15:29 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.48899999999999999,"15min":0.60899999999999999},"min":{"1min":0.23100000000000001,"5min":0.20599999999999999,"15min":0.187},"max":{"1min":2.9689999999999999,"5min":3.4239999999999999,"15min":12.382},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.44600000000000001,"15min":0.60799999999999998},"min":{"1min":0.221,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":2.706,"5min":3.8780000000000001,"15min":12.561},"last":0.28799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7184,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4851949568,"data_stored":5714546688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4702490624,"data_stored":5540323347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9552707584,"data_stored":11253137427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:16:12.584 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:16:12.584 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:16:32.586 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:16:33.185 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:16:33.187 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:16:33.217 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2945,"stamp":"2023-10-17T02:16:31.693123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29229255512,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2367223,"num_read_kb":9265581,"num_write":2340863,"num_write_kb":9363103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88328,"ondisk_log_size":88328,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":24092880,"kb_used_data":18722356,"kb_used_omap":1733,"kb_used_meta":5368762,"kb_avail":259022640,"statfs":{"total":289910292480,"available":265239183360,"internally_reserved":0,"allocated":19171692544,"data_stored":22571423459,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1774957,"internal_metadata":5497612947},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,2,1,7,8,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19243008,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4895,"num_read_kb":19544,"num_write":4753,"num_write_kb":19012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002006"},"pg_stats":[{"pgid":"2.7","version":"115'15420","reported_seq":21611,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:16:25.343743+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:16:25.343743+0000","last_peered":"2023-10-17T02:16:25.343743+0000","last_clean":"2023-10-17T02:16:25.343743+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:16:25.343743+0000","last_undegraded":"2023-10-17T02:16:25.343743+0000","last_fullsized":"2023-10-17T02:16:25.343743+0000","mapping_epoch":13,"log_start":"91'5400","ondisk_log_start":"91'5400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"115'11616","reported_seq":17436,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:16:25.343684+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:16:25.343684+0000","last_peered":"2023-10-17T02:16:25.343684+0000","last_clean":"2023-10-17T02:16:25.343684+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:16:25.343684+0000","last_undegraded":"2023-10-17T02:16:25.343684+0000","last_fullsized":"2023-10-17T02:16:25.343684+0000","mapping_epoch":13,"log_start":"58'1600","ondisk_log_start":"58'1600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"115'7788","reported_seq":10979,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:16:25.343133+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:16:25.343133+0000","last_peered":"2023-10-17T02:16:25.343133+0000","last_clean":"2023-10-17T02:16:25.343133+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:16:25.343133+0000","last_undegraded":"2023-10-17T02:16:25.343133+0000","last_fullsized":"2023-10-17T02:16:25.343133+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":7788,"log_dups_size":0,"ondisk_log_size":7788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"115'11653","reported_seq":16882,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:16:25.343454+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:16:25.343454+0000","last_peered":"2023-10-17T02:16:25.343454+0000","last_clean":"2023-10-17T02:16:25.343454+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:16:25.343454+0000","last_undegraded":"2023-10-17T02:16:25.343454+0000","last_fullsized":"2023-10-17T02:16:25.343454+0000","mapping_epoch":13,"log_start":"58'1600","ondisk_log_start":"58'1600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"115'596538","reported_seq":1189199,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:16:31.115436+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:16:31.115436+0000","last_peered":"2023-10-17T02:16:31.115436+0000","last_clean":"2023-10-17T02:16:31.115436+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:16:31.115436+0000","last_undegraded":"2023-10-17T02:16:31.115436+0000","last_fullsized":"2023-10-17T02:16:31.115436+0000","mapping_epoch":16,"log_start":"115'586510","ondisk_log_start":"115'586510","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7411380224,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":593337,"num_read_kb":2340056,"num_write":593781,"num_write_kb":2375124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":278,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:25.480115+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:14:25.480115+0000","last_peered":"2023-10-17T02:14:25.480115+0000","last_clean":"2023-10-17T02:14:25.480115+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:14:25.480115+0000","last_undegraded":"2023-10-17T02:14:25.480115+0000","last_fullsized":"2023-10-17T02:14:25.480115+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"115'573158","reported_seq":1143834,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:16:27.043059+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:16:27.043059+0000","last_peered":"2023-10-17T02:16:27.043059+0000","last_clean":"2023-10-17T02:16:27.043059+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:16:27.043059+0000","last_undegraded":"2023-10-17T02:16:27.043059+0000","last_fullsized":"2023-10-17T02:16:27.043059+0000","mapping_epoch":16,"log_start":"115'563119","ondisk_log_start":"115'563119","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":7123460096,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":571348,"num_read_kb":2255392,"num_write":570508,"num_write_kb":2282032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":3155,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:16:25.344212+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:16:25.344212+0000","last_peered":"2023-10-17T02:16:25.344212+0000","last_clean":"2023-10-17T02:16:25.344212+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:16:25.344212+0000","last_undegraded":"2023-10-17T02:16:25.344212+0000","last_fullsized":"2023-10-17T02:16:25.344212+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"115'597345","reported_seq":1192065,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:16:31.119487+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:16:31.119487+0000","last_peered":"2023-10-17T02:16:31.119487+0000","last_clean":"2023-10-17T02:16:31.119487+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:16:31.119487+0000","last_undegraded":"2023-10-17T02:16:31.119487+0000","last_fullsized":"2023-10-17T02:16:31.119487+0000","mapping_epoch":16,"log_start":"115'587325","ondisk_log_start":"115'587325","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7427878912,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":595428,"num_read_kb":2349228,"num_write":594591,"num_write_kb":2378364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4724,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:16:25.382382+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:16:25.382382+0000","last_peered":"2023-10-17T02:16:25.382382+0000","last_clean":"2023-10-17T02:16:25.382382+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:16:25.382382+0000","last_undegraded":"2023-10-17T02:16:25.382382+0000","last_fullsized":"2023-10-17T02:16:25.382382+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"115'583931","reported_seq":1166650,"reported_epoch":115,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:16:27.045559+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:16:27.045559+0000","last_peered":"2023-10-17T02:16:27.045559+0000","last_clean":"2023-10-17T02:16:27.045559+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:16:27.045559+0000","last_undegraded":"2023-10-17T02:16:27.045559+0000","last_fullsized":"2023-10-17T02:16:27.045559+0000","mapping_epoch":16,"log_start":"115'573918","ondisk_log_start":"115'573918","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2363,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 901s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7265943571,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":583148,"num_read_kb":2300252,"num_write":581353,"num_write_kb":2325406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"115'11624","reported_seq":16217,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:16:25.343353+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:16:25.343353+0000","last_peered":"2023-10-17T02:16:25.343353+0000","last_clean":"2023-10-17T02:16:25.343353+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:16:25.343353+0000","last_undegraded":"2023-10-17T02:16:25.343353+0000","last_fullsized":"2023-10-17T02:16:25.343353+0000","mapping_epoch":13,"log_start":"58'1600","ondisk_log_start":"58'1600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":298,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:12:45.653746+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:12:45.653746+0000","last_peered":"2023-10-17T02:12:45.653746+0000","last_clean":"2023-10-17T02:12:45.653746+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:12:45.653746+0000","last_undegraded":"2023-10-17T02:12:45.653746+0000","last_fullsized":"2023-10-17T02:12:45.653746+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29228662803,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2343261,"num_read_kb":9244928,"num_write":2340233,"num_write_kb":9360926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19169480704,"data_stored":22570340390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40100,"ondisk_log_size":40100,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34223,"internal_metadata":0},"log_size":48129,"ondisk_log_size":48129,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739503,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11137648,"kb_used_data":9359144,"kb_used_omap":854,"kb_used_meta":1777641,"kb_avail":83234192,"statfs":{"total":96636764160,"available":85231812608,"internally_reserved":0,"allocated":9583763456,"data_stored":11283542858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":875147,"internal_metadata":1820304757},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:15:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.42999999999999999,"15min":0.47299999999999998},"min":{"1min":0.187,"5min":0.17799999999999999,"15min":0.158},"max":{"1min":4.4450000000000003,"5min":4.4450000000000003,"15min":12.852},"last":5.7220000000000004},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.40300000000000002,"15min":0.47499999999999998},"min":{"1min":0.22,"5min":0.20300000000000001,"15min":0.192},"max":{"1min":3.2109999999999999,"5min":3.2109999999999999,"15min":12.784000000000001},"last":5.7560000000000002}]},{"osd":1,"last update":"Tue Oct 17 02:16:15 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.442,"15min":0.46800000000000003},"min":{"1min":0.215,"5min":0.215,"15min":0.20899999999999999},"max":{"1min":2.9020000000000001,"5min":3.302,"15min":6.202},"last":0.22},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.46600000000000003,"15min":0.56499999999999995},"min":{"1min":0.26500000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.9380000000000002,"5min":2.9380000000000002,"15min":12.824999999999999},"last":0.29099999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739499,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6686896,"kb_used_data":4608676,"kb_used_omap":423,"kb_used_meta":2077784,"kb_avail":87684944,"statfs":{"total":96636764160,"available":89789382656,"internally_reserved":0,"allocated":4719284224,"data_stored":5556837225,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":434080,"internal_metadata":2127650912},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,1,7,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:15:41 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.57599999999999996,"15min":0.63400000000000001},"min":{"1min":0.20200000000000001,"5min":0.17999999999999999,"15min":0.16600000000000001},"max":{"1min":0.61899999999999999,"5min":5.1109999999999998,"15min":11.891999999999999},"last":0.314},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.623,"15min":0.68100000000000005},"min":{"1min":0.30499999999999999,"5min":0.24099999999999999,"15min":0.224},"max":{"1min":0.64600000000000002,"5min":5.1449999999999996,"15min":11.926},"last":0.35299999999999998}]},{"osd":2,"last update":"Tue Oct 17 02:15:47 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.745,"15min":0.751},"min":{"1min":0.23200000000000001,"5min":0.20599999999999999,"15min":0.19500000000000001},"max":{"1min":0.54600000000000004,"5min":13.300000000000001,"15min":25.128},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.45200000000000001,"15min":0.48699999999999999},"min":{"1min":0.245,"5min":0.219,"15min":0.214},"max":{"1min":0.57899999999999996,"5min":3.597,"15min":6.7800000000000002},"last":0.28799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739500,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6268336,"kb_used_data":4754536,"kb_used_omap":454,"kb_used_meta":1513337,"kb_avail":88103504,"statfs":{"total":96636764160,"available":90217988096,"internally_reserved":0,"allocated":4868644864,"data_stored":5731043376,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":465730,"internal_metadata":1549657278},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:16:26 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.85799999999999998,"15min":0.624},"min":{"1min":0.16900000000000001,"5min":0.161,"15min":0.161},"max":{"1min":5.4480000000000004,"5min":13.834,"15min":13.834},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.56499999999999995},"min":{"1min":0.20499999999999999,"5min":0.185,"15min":0.185},"max":{"1min":4.6900000000000004,"5min":9.0730000000000004,"15min":9.0730000000000004},"last":0.252}]},{"osd":2,"last update":"Tue Oct 17 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.53600000000000003,"15min":0.60899999999999999},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.2300000000000004,"5min":5.2300000000000004,"15min":12.382},"last":5.2300000000000004},{"interface":"front","average":{"1min":0.752,"5min":0.49299999999999999,"15min":0.60299999999999998},"min":{"1min":0.23100000000000001,"5min":0.221,"15min":0.20200000000000001},"max":{"1min":5.3529999999999998,"5min":5.3529999999999998,"15min":12.561},"last":5.3529999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7184,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4867579904,"data_stored":5730177024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4718022656,"data_stored":5555855379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9583878144,"data_stored":11284307987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:16:33.219 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:16:33.219 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:16:45.301 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:16:45.285+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T02:16:45.301 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:16:45.289+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T02:16:45.302 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:16:45.293+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T02:16:46.606 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:16:46.593+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbbb2b0 2023-10-17T02:16:46.623 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:16:46.613+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T02:16:46.624 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:16:46.613+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T02:16:46.624 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:16:46.613+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T02:16:46.624 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:16:46.621+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T02:16:53.221 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:16:53.730 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:16:53.731 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:16:53.764 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2956,"stamp":"2023-10-17T02:16:53.696735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29263088472,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2375611,"num_read_kb":9298969,"num_write":2349190,"num_write_kb":9396411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88684,"ondisk_log_size":88684,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":24389740,"kb_used_data":18793424,"kb_used_omap":1733,"kb_used_meta":5594554,"kb_avail":258725780,"statfs":{"total":289910292480,"available":264935198720,"internally_reserved":0,"allocated":19244466176,"data_stored":22644197091,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1774957,"internal_metadata":5728823955},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,9,12,12,14,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18763776,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4645,"num_read_kb":18472,"num_write":4606,"num_write_kb":18424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001985"},"pg_stats":[{"pgid":"2.7","version":"115'15497","reported_seq":21708,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:16:50.943677+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:16:50.943677+0000","last_peered":"2023-10-17T02:16:50.943677+0000","last_clean":"2023-10-17T02:16:50.943677+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:16:50.943677+0000","last_undegraded":"2023-10-17T02:16:50.943677+0000","last_fullsized":"2023-10-17T02:16:50.943677+0000","mapping_epoch":13,"log_start":"91'5400","ondisk_log_start":"91'5400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"115'11674","reported_seq":17509,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:16:50.943759+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:16:50.943759+0000","last_peered":"2023-10-17T02:16:50.943759+0000","last_clean":"2023-10-17T02:16:50.943759+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:16:50.943759+0000","last_undegraded":"2023-10-17T02:16:50.943759+0000","last_fullsized":"2023-10-17T02:16:50.943759+0000","mapping_epoch":13,"log_start":"58'1600","ondisk_log_start":"58'1600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"115'7828","reported_seq":11030,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:16:50.943663+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:16:50.943663+0000","last_peered":"2023-10-17T02:16:50.943663+0000","last_clean":"2023-10-17T02:16:50.943663+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:16:50.943663+0000","last_undegraded":"2023-10-17T02:16:50.943663+0000","last_fullsized":"2023-10-17T02:16:50.943663+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":7828,"log_dups_size":0,"ondisk_log_size":7828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"115'11712","reported_seq":16957,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:16:50.943448+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:16:50.943448+0000","last_peered":"2023-10-17T02:16:50.943448+0000","last_clean":"2023-10-17T02:16:50.943448+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:16:50.943448+0000","last_undegraded":"2023-10-17T02:16:50.943448+0000","last_fullsized":"2023-10-17T02:16:50.943448+0000","mapping_epoch":13,"log_start":"58'1700","ondisk_log_start":"58'1700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"115'598369","reported_seq":1192941,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:16:51.247497+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:16:51.247497+0000","last_peered":"2023-10-17T02:16:51.247497+0000","last_clean":"2023-10-17T02:16:51.247497+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:16:51.247497+0000","last_undegraded":"2023-10-17T02:16:51.247497+0000","last_fullsized":"2023-10-17T02:16:51.247497+0000","mapping_epoch":16,"log_start":"115'588310","ondisk_log_start":"115'588310","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7418802176,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":595247,"num_read_kb":2347668,"num_write":595612,"num_write_kb":2382448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":278,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:25.480115+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:14:25.480115+0000","last_peered":"2023-10-17T02:14:25.480115+0000","last_clean":"2023-10-17T02:14:25.480115+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:14:25.480115+0000","last_undegraded":"2023-10-17T02:14:25.480115+0000","last_fullsized":"2023-10-17T02:14:25.480115+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"115'575491","reported_seq":1148429,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:16:52.190671+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:16:52.190671+0000","last_peered":"2023-10-17T02:16:52.190671+0000","last_clean":"2023-10-17T02:16:52.190671+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:16:52.190671+0000","last_undegraded":"2023-10-17T02:16:52.190671+0000","last_fullsized":"2023-10-17T02:16:52.190671+0000","mapping_epoch":16,"log_start":"115'565419","ondisk_log_start":"115'565419","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":7132938240,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":573618,"num_read_kb":2264424,"num_write":572841,"num_write_kb":2291364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":3165,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:16:50.943875+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:16:50.943875+0000","last_peered":"2023-10-17T02:16:50.943875+0000","last_clean":"2023-10-17T02:16:50.943875+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:16:50.943875+0000","last_undegraded":"2023-10-17T02:16:50.943875+0000","last_fullsized":"2023-10-17T02:16:50.943875+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"115'599197","reported_seq":1195800,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:16:51.256824+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:16:51.256824+0000","last_peered":"2023-10-17T02:16:51.256824+0000","last_clean":"2023-10-17T02:16:51.256824+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:16:51.256824+0000","last_undegraded":"2023-10-17T02:16:51.256824+0000","last_fullsized":"2023-10-17T02:16:51.256824+0000","mapping_epoch":16,"log_start":"115'589125","ondisk_log_start":"115'589125","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7435427840,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":597313,"num_read_kb":2356724,"num_write":596443,"num_write_kb":2385772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4740,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:16:50.945479+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:16:50.945479+0000","last_peered":"2023-10-17T02:16:50.945479+0000","last_clean":"2023-10-17T02:16:50.945479+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:16:50.945479+0000","last_undegraded":"2023-10-17T02:16:50.945479+0000","last_fullsized":"2023-10-17T02:16:50.945479+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"115'586278","reported_seq":1171332,"reported_epoch":115,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:16:52.193961+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:16:52.193961+0000","last_peered":"2023-10-17T02:16:52.193961+0000","last_clean":"2023-10-17T02:16:52.193961+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:16:52.193961+0000","last_undegraded":"2023-10-17T02:16:52.193961+0000","last_fullsized":"2023-10-17T02:16:52.193961+0000","mapping_epoch":16,"log_start":"115'576218","ondisk_log_start":"115'576218","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2433,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 926s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7275327507,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":585471,"num_read_kb":2309500,"num_write":583664,"num_write_kb":2334650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"115'11683","reported_seq":16292,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:16:50.943799+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:16:50.943799+0000","last_peered":"2023-10-17T02:16:50.943799+0000","last_clean":"2023-10-17T02:16:50.943799+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:16:50.943799+0000","last_undegraded":"2023-10-17T02:16:50.943799+0000","last_fullsized":"2023-10-17T02:16:50.943799+0000","mapping_epoch":13,"log_start":"58'1600","ondisk_log_start":"58'1600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":299,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:46.423633+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:14:46.423633+0000","last_peered":"2023-10-17T02:14:46.423633+0000","last_clean":"2023-10-17T02:14:46.423633+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:14:46.423633+0000","last_undegraded":"2023-10-17T02:14:46.423633+0000","last_fullsized":"2023-10-17T02:14:46.423633+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29262495763,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2351649,"num_read_kb":9278316,"num_write":2348560,"num_write_kb":9394234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19241431040,"data_stored":22642290726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40263,"ondisk_log_size":40263,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34223,"internal_metadata":0},"log_size":48322,"ondisk_log_size":48322,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739508,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11295000,"kb_used_data":9397004,"kb_used_omap":854,"kb_used_meta":1897129,"kb_avail":83076840,"statfs":{"total":96636764160,"available":85070684160,"internally_reserved":0,"allocated":9622532096,"data_stored":11322311498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":875147,"internal_metadata":1942660469},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,1,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:16:43 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.46200000000000002,"15min":0.49099999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.158},"max":{"1min":5.7220000000000004,"5min":5.7220000000000004,"15min":12.852},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.44600000000000001,"15min":0.48699999999999999},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.192},"max":{"1min":5.7560000000000002,"5min":5.7560000000000002,"15min":12.784000000000001},"last":0.29999999999999999}]},{"osd":1,"last update":"Tue Oct 17 02:16:15 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.442,"15min":0.46800000000000003},"min":{"1min":0.215,"5min":0.215,"15min":0.20899999999999999},"max":{"1min":2.9020000000000001,"5min":3.302,"15min":6.202},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.46600000000000003,"15min":0.56499999999999995},"min":{"1min":0.26500000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.9380000000000002,"5min":2.9380000000000002,"15min":12.824999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739504,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6768216,"kb_used_data":4627216,"kb_used_omap":423,"kb_used_meta":2140568,"kb_avail":87603624,"statfs":{"total":96636764160,"available":89706110976,"internally_reserved":0,"allocated":4738269184,"data_stored":5575822185,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":434080,"internal_metadata":2191941728},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,5,5,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:16:42 2023","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.51600000000000001,"15min":0.628},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.16900000000000001},"max":{"1min":0.54200000000000004,"5min":5.1109999999999998,"15min":11.891999999999999},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.61699999999999999,"15min":0.69199999999999995},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.224},"max":{"1min":5.0350000000000001,"5min":5.1449999999999996,"15min":11.926},"last":0.376}]},{"osd":2,"last update":"Tue Oct 17 02:16:48 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.64600000000000002,"15min":0.71699999999999997},"min":{"1min":0.20899999999999999,"5min":0.20599999999999999,"15min":0.19500000000000001},"max":{"1min":0.96599999999999997,"5min":13.300000000000001,"15min":25.128},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.50700000000000001,"15min":0.47599999999999998},"min":{"1min":0.26100000000000001,"5min":0.23000000000000001,"15min":0.214},"max":{"1min":4.9980000000000002,"5min":4.9980000000000002,"15min":5.2709999999999999},"last":0.39100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739504,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6326524,"kb_used_data":4769204,"kb_used_omap":454,"kb_used_meta":1556857,"kb_avail":88045316,"statfs":{"total":96636764160,"available":90158403584,"internally_reserved":0,"allocated":4883664896,"data_stored":5746063408,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":465730,"internal_metadata":1594221758},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,6,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:16:26 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.85799999999999998,"15min":0.624},"min":{"1min":0.16900000000000001,"5min":0.161,"15min":0.161},"max":{"1min":5.4480000000000004,"5min":13.834,"15min":13.834},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.56499999999999995},"min":{"1min":0.20499999999999999,"5min":0.185,"15min":0.185},"max":{"1min":4.6900000000000004,"5min":9.0730000000000004,"15min":9.0730000000000004},"last":0.23200000000000001}]},{"osd":2,"last update":"Tue Oct 17 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.53600000000000003,"15min":0.60899999999999999},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.2300000000000004,"5min":5.2300000000000004,"15min":12.382},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.752,"5min":0.49299999999999999,"15min":0.60299999999999998},"min":{"1min":0.23100000000000001,"5min":0.221,"15min":0.20200000000000001},"max":{"1min":5.3529999999999998,"5min":5.3529999999999998,"15min":12.561},"last":0.27300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7184,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4882673664,"data_stored":5745270784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4737048576,"data_stored":5574881299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9621708800,"data_stored":11322138643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:16:53.766 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:16:53.767 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:17:12.698 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:17:12.693+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fbd6680 2023-10-17T02:17:12.703 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:17:12.701+0000 7f31dd33a640 -1 reset not still connected to 0x55e90fea4820 2023-10-17T02:17:12.728 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:17:12.725+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T02:17:12.731 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:17:12.729+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T02:17:13.768 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:17:14.281 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:17:14.282 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:17:14.317 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2966,"stamp":"2023-10-17T02:17:13.700117+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29293357912,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2383033,"num_read_kb":9328553,"num_write":2356606,"num_write_kb":9426075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88437,"ondisk_log_size":88437,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":24649244,"kb_used_data":18852548,"kb_used_omap":1733,"kb_used_meta":5794938,"kb_avail":258466276,"statfs":{"total":289910292480,"available":264669466624,"internally_reserved":0,"allocated":19305009152,"data_stored":22704740067,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1774957,"internal_metadata":5934017171},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,6,10,16,14,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19279872,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4632,"num_read_kb":18484,"num_write":4723,"num_write_kb":18892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002077"},"pg_stats":[{"pgid":"2.7","version":"115'15521","reported_seq":21748,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:17:11.423812+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:17:11.423812+0000","last_peered":"2023-10-17T02:17:11.423812+0000","last_clean":"2023-10-17T02:17:11.423812+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:17:11.423812+0000","last_undegraded":"2023-10-17T02:17:11.423812+0000","last_fullsized":"2023-10-17T02:17:11.423812+0000","mapping_epoch":13,"log_start":"91'5500","ondisk_log_start":"91'5500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"115'11692","reported_seq":17539,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:17:11.423466+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:17:11.423466+0000","last_peered":"2023-10-17T02:17:11.423466+0000","last_clean":"2023-10-17T02:17:11.423466+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:17:11.423466+0000","last_undegraded":"2023-10-17T02:17:11.423466+0000","last_fullsized":"2023-10-17T02:17:11.423466+0000","mapping_epoch":13,"log_start":"58'1600","ondisk_log_start":"58'1600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"115'7840","reported_seq":11050,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:17:11.423159+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:17:11.423159+0000","last_peered":"2023-10-17T02:17:11.423159+0000","last_clean":"2023-10-17T02:17:11.423159+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:17:11.423159+0000","last_undegraded":"2023-10-17T02:17:11.423159+0000","last_fullsized":"2023-10-17T02:17:11.423159+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":7840,"log_dups_size":0,"ondisk_log_size":7840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"115'11730","reported_seq":16987,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:17:11.423444+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:17:11.423444+0000","last_peered":"2023-10-17T02:17:11.423444+0000","last_clean":"2023-10-17T02:17:11.423444+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:17:11.423444+0000","last_undegraded":"2023-10-17T02:17:11.423444+0000","last_fullsized":"2023-10-17T02:17:11.423444+0000","mapping_epoch":13,"log_start":"58'1700","ondisk_log_start":"58'1700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"115'600333","reported_seq":1196833,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:17:11.402578+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:17:11.402578+0000","last_peered":"2023-10-17T02:17:11.402578+0000","last_clean":"2023-10-17T02:17:11.402578+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:17:11.402578+0000","last_undegraded":"2023-10-17T02:17:11.402578+0000","last_fullsized":"2023-10-17T02:17:11.402578+0000","mapping_epoch":16,"log_start":"115'590310","ondisk_log_start":"115'590310","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7426793472,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":597175,"num_read_kb":2355368,"num_write":597576,"num_write_kb":2390304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":278,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:25.480115+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:14:25.480115+0000","last_peered":"2023-10-17T02:14:25.480115+0000","last_clean":"2023-10-17T02:14:25.480115+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:14:25.480115+0000","last_undegraded":"2023-10-17T02:14:25.480115+0000","last_fullsized":"2023-10-17T02:14:25.480115+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"115'577258","reported_seq":1151981,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:17:12.296468+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:17:12.296468+0000","last_peered":"2023-10-17T02:17:12.296468+0000","last_clean":"2023-10-17T02:17:12.296468+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:17:12.296468+0000","last_undegraded":"2023-10-17T02:17:12.296468+0000","last_fullsized":"2023-10-17T02:17:12.296468+0000","mapping_epoch":16,"log_start":"115'567219","ondisk_log_start":"115'567219","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":7140151296,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":575396,"num_read_kb":2271500,"num_write":574608,"num_write_kb":2298432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'70","reported_seq":3173,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:17:11.423327+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:17:11.423327+0000","last_peered":"2023-10-17T02:17:11.423327+0000","last_clean":"2023-10-17T02:17:11.423327+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:17:11.423327+0000","last_undegraded":"2023-10-17T02:17:11.423327+0000","last_fullsized":"2023-10-17T02:17:11.423327+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"115'601057","reported_seq":1199467,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:17:11.410963+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:17:11.410963+0000","last_peered":"2023-10-17T02:17:11.410963+0000","last_clean":"2023-10-17T02:17:11.410963+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:17:11.410963+0000","last_undegraded":"2023-10-17T02:17:11.410963+0000","last_fullsized":"2023-10-17T02:17:11.410963+0000","mapping_epoch":16,"log_start":"115'591025","ondisk_log_start":"115'591025","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7443042304,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":599122,"num_read_kb":2363928,"num_write":598303,"num_write_kb":2393212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'113","reported_seq":4752,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:17:11.442280+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:17:11.442280+0000","last_peered":"2023-10-17T02:17:11.442280+0000","last_clean":"2023-10-17T02:17:11.442280+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:17:11.442280+0000","last_undegraded":"2023-10-17T02:17:11.442280+0000","last_fullsized":"2023-10-17T02:17:11.442280+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"115'588150","reported_seq":1175117,"reported_epoch":115,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-17T02:17:12.308326+0000","last_change":"2023-10-17T02:01:25.395674+0000","last_active":"2023-10-17T02:17:12.308326+0000","last_peered":"2023-10-17T02:17:12.308326+0000","last_clean":"2023-10-17T02:17:12.308326+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:17:12.308326+0000","last_undegraded":"2023-10-17T02:17:12.308326+0000","last_fullsized":"2023-10-17T02:17:12.308326+0000","mapping_epoch":16,"log_start":"115'578118","ondisk_log_start":"115'578118","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T00:42:17.806712+0000","last_clean_scrub_stamp":"2023-10-17T00:42:17.806712+0000","objects_scrubbed":2511,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 946s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7282778131,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":587378,"num_read_kb":2317104,"num_write":585489,"num_write_kb":2341950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"115'11701","reported_seq":16322,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:17:11.423322+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:17:11.423322+0000","last_peered":"2023-10-17T02:17:11.423322+0000","last_clean":"2023-10-17T02:17:11.423322+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:17:11.423322+0000","last_undegraded":"2023-10-17T02:17:11.423322+0000","last_fullsized":"2023-10-17T02:17:11.423322+0000","mapping_epoch":13,"log_start":"58'1700","ondisk_log_start":"58'1700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":299,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:46.423633+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:14:46.423633+0000","last_peered":"2023-10-17T02:14:46.423633+0000","last_clean":"2023-10-17T02:14:46.423633+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:14:46.423633+0000","last_undegraded":"2023-10-17T02:14:46.423633+0000","last_fullsized":"2023-10-17T02:14:46.423633+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29292765203,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2359071,"num_read_kb":9307900,"num_write":2355976,"num_write_kb":9423898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19302236160,"data_stored":22703095846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40126,"ondisk_log_size":40126,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34223,"internal_metadata":0},"log_size":48212,"ondisk_log_size":48212,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739512,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11464824,"kb_used_data":9426736,"kb_used_omap":854,"kb_used_meta":2037225,"kb_avail":82907016,"statfs":{"total":96636764160,"available":84896784384,"internally_reserved":0,"allocated":9652977664,"data_stored":11352757066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":875147,"internal_metadata":2086118773},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,2,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:16:43 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.46200000000000002,"15min":0.49099999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.158},"max":{"1min":5.7220000000000004,"5min":5.7220000000000004,"15min":12.852},"last":0.255},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.44600000000000001,"15min":0.48699999999999999},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.192},"max":{"1min":5.7560000000000002,"5min":5.7560000000000002,"15min":12.784000000000001},"last":0.28899999999999998}]},{"osd":1,"last update":"Tue Oct 17 02:16:15 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.442,"15min":0.46800000000000003},"min":{"1min":0.215,"5min":0.215,"15min":0.20899999999999999},"max":{"1min":2.9020000000000001,"5min":3.302,"15min":6.202},"last":0.375},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.46600000000000003,"15min":0.56499999999999995},"min":{"1min":0.26500000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.9380000000000002,"5min":2.9380000000000002,"15min":12.824999999999999},"last":0.40999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739508,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6764416,"kb_used_data":4641400,"kb_used_omap":423,"kb_used_meta":2122584,"kb_avail":87607424,"statfs":{"total":96636764160,"available":89710002176,"internally_reserved":0,"allocated":4752793600,"data_stored":5590346601,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":434080,"internal_metadata":2173526112},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,4,7,8,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:16:42 2023","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.51600000000000001,"15min":0.628},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.16900000000000001},"max":{"1min":0.54200000000000004,"5min":5.1109999999999998,"15min":11.891999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.61699999999999999,"15min":0.69199999999999995},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.224},"max":{"1min":5.0350000000000001,"5min":5.1449999999999996,"15min":11.926},"last":0.77400000000000002}]},{"osd":2,"last update":"Tue Oct 17 02:16:48 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.64600000000000002,"15min":0.71699999999999997},"min":{"1min":0.20899999999999999,"5min":0.20599999999999999,"15min":0.19500000000000001},"max":{"1min":0.96599999999999997,"5min":13.300000000000001,"15min":25.128},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.50700000000000001,"15min":0.47599999999999998},"min":{"1min":0.26100000000000001,"5min":0.23000000000000001,"15min":0.214},"max":{"1min":4.9980000000000002,"5min":4.9980000000000002,"15min":5.2709999999999999},"last":0.70099999999999996}]}]},{"osd":0,"up_from":8,"seq":34359739508,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6420004,"kb_used_data":4784412,"kb_used_omap":454,"kb_used_meta":1635129,"kb_avail":87951836,"statfs":{"total":96636764160,"available":90062680064,"internally_reserved":0,"allocated":4899237888,"data_stored":5761636400,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":465730,"internal_metadata":1674372286},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,4,6,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:16:26 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.85799999999999998,"15min":0.624},"min":{"1min":0.16900000000000001,"5min":0.161,"15min":0.161},"max":{"1min":5.4480000000000004,"5min":13.834,"15min":13.834},"last":0.255},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.56499999999999995},"min":{"1min":0.20499999999999999,"5min":0.185,"15min":0.185},"max":{"1min":4.6900000000000004,"5min":9.0730000000000004,"15min":9.0730000000000004},"last":0.375}]},{"osd":2,"last update":"Tue Oct 17 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.53600000000000003,"15min":0.60899999999999999},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.2300000000000004,"5min":5.2300000000000004,"15min":12.382},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.752,"5min":0.49299999999999999,"15min":0.60299999999999998},"min":{"1min":0.23100000000000001,"5min":0.221,"15min":0.20200000000000001},"max":{"1min":5.3529999999999998,"5min":5.3529999999999998,"15min":12.561},"last":0.34000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7184,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4898328576,"data_stored":5760925696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4751761408,"data_stored":5589594131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9652146176,"data_stored":11352576019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:17:14.319 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-17T00:42:17.806712+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=0, tm_min=42, tm_sec=17, tm_wday=1, tm_yday=290, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=17, tm_hour=1, tm_min=48, tm_sec=0, tm_wday=1, tm_yday=290, tm_isdst=0) 2023-10-17T02:17:14.319 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-17T02:17:20.542 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:17:20.536+0000 7f31dd33a640 -1 reset not still connected to 0x55e9102d9380 2023-10-17T02:17:20.545 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:17:20.540+0000 7f31dd33a640 -1 reset not still connected to 0x55e91039b6c0 2023-10-17T02:17:20.556 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:17:20.552+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a700d0 2023-10-17T02:17:20.567 INFO:tasks.ceph.osd.1.smithi169.stderr:2023-10-17T02:17:20.564+0000 7f31dd33a640 -1 reset not still connected to 0x55e910a79a00 2023-10-17T02:17:34.320 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:17:34.920 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-10-17T02:17:34.921 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2023-10-17T02:17:34.965 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":2976,"stamp":"2023-10-17T02:17:33.703817+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29324585816,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2390445,"num_read_kb":9358133,"num_write":2364267,"num_write_kb":9456719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88705,"ondisk_log_size":88705,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23895780,"kb_used_data":18913932,"kb_used_omap":1734,"kb_used_meta":4980089,"kb_avail":259219740,"statfs":{"total":289910292480,"available":265441013760,"internally_reserved":0,"allocated":19367866368,"data_stored":22767597283,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1776636,"internal_metadata":5099611140},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,5,11,19,12,16,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19648512,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4652,"num_read_kb":18576,"num_write":4824,"num_write_kb":19296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002338"},"pg_stats":[{"pgid":"2.7","version":"115'15595","reported_seq":21838,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:17:31.903703+0000","last_change":"2023-10-17T01:48:19.607569+0000","last_active":"2023-10-17T02:17:31.903703+0000","last_peered":"2023-10-17T02:17:31.903703+0000","last_clean":"2023-10-17T02:17:31.903703+0000","last_became_active":"2023-10-17T00:42:12.797805+0000","last_became_peered":"2023-10-17T00:42:12.797805+0000","last_unstale":"2023-10-17T02:17:31.903703+0000","last_undegraded":"2023-10-17T02:17:31.903703+0000","last_fullsized":"2023-10-17T02:17:31.903703+0000","mapping_epoch":13,"log_start":"91'5500","ondisk_log_start":"91'5500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'11246","last_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_deep_scrub":"95'11246","last_deep_scrub_stamp":"2023-10-17T01:48:19.607371+0000","last_clean_scrub_stamp":"2023-10-17T01:48:19.607371+0000","objects_scrubbed":10,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:21:03.965729+0000","scrub_duration":0.097090493999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4215,"num_read_kb":3573,"num_write":92,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2072,"num_omap_keys":44,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"115'11749","reported_seq":17608,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:17:31.904066+0000","last_change":"2023-10-17T01:48:18.640210+0000","last_active":"2023-10-17T02:17:31.904066+0000","last_peered":"2023-10-17T02:17:31.904066+0000","last_clean":"2023-10-17T02:17:31.904066+0000","last_became_active":"2023-10-17T00:42:12.799545+0000","last_became_peered":"2023-10-17T00:42:12.799545+0000","last_unstale":"2023-10-17T02:17:31.904066+0000","last_undegraded":"2023-10-17T02:17:31.904066+0000","last_fullsized":"2023-10-17T02:17:31.904066+0000","mapping_epoch":13,"log_start":"58'1700","ondisk_log_start":"58'1700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8473","last_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_deep_scrub":"95'8473","last_deep_scrub_stamp":"2023-10-17T01:48:18.640014+0000","last_clean_scrub_stamp":"2023-10-17T01:48:18.640014+0000","objects_scrubbed":10,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:28.124980+0000","scrub_duration":0.093909120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5323,"num_read_kb":4833,"num_write":105,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2965,"num_omap_keys":76,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"115'7880","reported_seq":11098,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:17:31.903442+0000","last_change":"2023-10-17T01:48:14.603176+0000","last_active":"2023-10-17T02:17:31.903442+0000","last_peered":"2023-10-17T02:17:31.903442+0000","last_clean":"2023-10-17T02:17:31.903442+0000","last_became_active":"2023-10-17T00:42:12.797857+0000","last_became_peered":"2023-10-17T00:42:12.797857+0000","last_unstale":"2023-10-17T02:17:31.903442+0000","last_undegraded":"2023-10-17T02:17:31.903442+0000","last_fullsized":"2023-10-17T02:17:31.903442+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'5677","last_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_deep_scrub":"95'5677","last_deep_scrub_stamp":"2023-10-17T01:48:14.602998+0000","last_clean_scrub_stamp":"2023-10-17T01:48:14.602998+0000","objects_scrubbed":7,"log_size":7880,"log_dups_size":0,"ondisk_log_size":7880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T12:45:46.717773+0000","scrub_duration":0.045625590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2135,"num_read_kb":1795,"num_write":45,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"115'11786","reported_seq":17055,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:17:31.903364+0000","last_change":"2023-10-17T01:48:16.736398+0000","last_active":"2023-10-17T02:17:31.903364+0000","last_peered":"2023-10-17T02:17:31.903364+0000","last_clean":"2023-10-17T02:17:31.903364+0000","last_became_active":"2023-10-17T00:42:12.796419+0000","last_became_peered":"2023-10-17T00:42:12.796419+0000","last_unstale":"2023-10-17T02:17:31.903364+0000","last_undegraded":"2023-10-17T02:17:31.903364+0000","last_fullsized":"2023-10-17T02:17:31.903364+0000","mapping_epoch":13,"log_start":"58'1700","ondisk_log_start":"58'1700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8499","last_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_deep_scrub":"95'8499","last_deep_scrub_stamp":"2023-10-17T01:48:16.736165+0000","last_clean_scrub_stamp":"2023-10-17T01:48:16.736165+0000","objects_scrubbed":15,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T13:25:43.928052+0000","scrub_duration":0.123789713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3786,"num_read_kb":3256,"num_write":81,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2084,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"115'602287","reported_seq":1200712,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:17:31.535810+0000","last_change":"2023-10-17T02:05:04.847963+0000","last_active":"2023-10-17T02:17:31.535810+0000","last_peered":"2023-10-17T02:17:31.535810+0000","last_clean":"2023-10-17T02:17:31.535810+0000","last_became_active":"2023-10-17T00:42:19.130459+0000","last_became_peered":"2023-10-17T00:42:19.130459+0000","last_unstale":"2023-10-17T02:17:31.535810+0000","last_undegraded":"2023-10-17T02:17:31.535810+0000","last_fullsized":"2023-10-17T02:17:31.535810+0000","mapping_epoch":16,"log_start":"115'592210","ondisk_log_start":"115'592210","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'531743","last_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_deep_scrub":"109'531743","last_deep_scrub_stamp":"2023-10-17T02:05:04.847709+0000","last_clean_scrub_stamp":"2023-10-17T02:05:04.847709+0000","objects_scrubbed":2596,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1012,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T11:34:58.567145+0000","scrub_duration":1011.673934227,"objects_trimmed":0,"snaptrim_duration":0.00082222999999999997,"stat_sum":{"num_bytes":7434768384,"num_objects":2596,"num_object_clones":1298,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":599102,"num_read_kb":2363064,"num_write":599530,"num_write_kb":2398120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'45","reported_seq":278,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:25.480115+0000","last_change":"2023-10-17T01:48:03.009614+0000","last_active":"2023-10-17T02:14:25.480115+0000","last_peered":"2023-10-17T02:14:25.480115+0000","last_clean":"2023-10-17T02:14:25.480115+0000","last_became_active":"2023-10-17T00:42:12.796730+0000","last_became_peered":"2023-10-17T00:42:12.796730+0000","last_unstale":"2023-10-17T02:14:25.480115+0000","last_undegraded":"2023-10-17T02:14:25.480115+0000","last_fullsized":"2023-10-17T02:14:25.480115+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'45","last_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_deep_scrub":"95'45","last_deep_scrub_stamp":"2023-10-17T01:48:03.009394+0000","last_clean_scrub_stamp":"2023-10-17T01:48:03.009394+0000","objects_scrubbed":13,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:07:24.274719+0000","scrub_duration":0.079157687000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":87,"num_read_kb":84,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"115'579067","reported_seq":1155563,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:17:32.395515+0000","last_change":"2023-10-17T02:01:25.110421+0000","last_active":"2023-10-17T02:17:32.395515+0000","last_peered":"2023-10-17T02:17:32.395515+0000","last_clean":"2023-10-17T02:17:32.395515+0000","last_became_active":"2023-10-17T00:42:19.130449+0000","last_became_peered":"2023-10-17T00:42:19.130449+0000","last_unstale":"2023-10-17T02:17:32.395515+0000","last_undegraded":"2023-10-17T02:17:32.395515+0000","last_fullsized":"2023-10-17T02:17:32.395515+0000","mapping_epoch":16,"log_start":"115'569019","ondisk_log_start":"115'569019","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'492031","last_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_deep_scrub":"106'492031","last_deep_scrub_stamp":"2023-10-17T02:01:25.110164+0000","last_clean_scrub_stamp":"2023-10-17T02:01:25.110164+0000","objects_scrubbed":2496,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":784,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:49:39.245331+0000","scrub_duration":783.55928450099998,"objects_trimmed":0,"snaptrim_duration":0.00087589600000000003,"stat_sum":{"num_bytes":7147520000,"num_objects":2496,"num_object_clones":1248,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":577168,"num_read_kb":2278580,"num_write":576417,"num_write_kb":2305668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"115'72","reported_seq":3181,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:17:31.903291+0000","last_change":"2023-10-17T01:48:09.010864+0000","last_active":"2023-10-17T02:17:31.903291+0000","last_peered":"2023-10-17T02:17:31.903291+0000","last_clean":"2023-10-17T02:17:31.903291+0000","last_became_active":"2023-10-17T00:42:12.798608+0000","last_became_peered":"2023-10-17T00:42:12.798608+0000","last_unstale":"2023-10-17T02:17:31.903291+0000","last_undegraded":"2023-10-17T02:17:31.903291+0000","last_fullsized":"2023-10-17T02:17:31.903291+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'70","last_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_deep_scrub":"95'70","last_deep_scrub_stamp":"2023-10-17T01:48:09.010130+0000","last_clean_scrub_stamp":"2023-10-17T01:48:09.010130+0000","objects_scrubbed":9,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:09.362292+0000","scrub_duration":0.14101438899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2021,"num_read_kb":1712,"num_write":35,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"115'603057","reported_seq":1203379,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:17:31.549172+0000","last_change":"2023-10-17T02:02:52.311041+0000","last_active":"2023-10-17T02:17:31.549172+0000","last_peered":"2023-10-17T02:17:31.549172+0000","last_clean":"2023-10-17T02:17:31.549172+0000","last_became_active":"2023-10-17T00:42:19.130348+0000","last_became_peered":"2023-10-17T00:42:19.130348+0000","last_unstale":"2023-10-17T02:17:31.549172+0000","last_undegraded":"2023-10-17T02:17:31.549172+0000","last_fullsized":"2023-10-17T02:17:31.549172+0000","mapping_epoch":16,"log_start":"115'593025","ondisk_log_start":"115'593025","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'520559","last_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_deep_scrub":"107'520559","last_deep_scrub_stamp":"2023-10-17T02:02:52.310444+0000","last_clean_scrub_stamp":"2023-10-17T02:02:52.310444+0000","objects_scrubbed":2602,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":888,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:33:52.189682+0000","scrub_duration":887.209537673,"objects_trimmed":0,"snaptrim_duration":0.0012273340000000001,"stat_sum":{"num_bytes":7451197440,"num_objects":2602,"num_object_clones":1301,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":601034,"num_read_kb":2371552,"num_write":600303,"num_write_kb":2401212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"115'116","reported_seq":4765,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:17:31.916470+0000","last_change":"2023-10-17T01:48:04.917667+0000","last_active":"2023-10-17T02:17:31.916470+0000","last_peered":"2023-10-17T02:17:31.916470+0000","last_clean":"2023-10-17T02:17:31.916470+0000","last_became_active":"2023-10-17T00:42:12.797946+0000","last_became_peered":"2023-10-17T00:42:12.797946+0000","last_unstale":"2023-10-17T02:17:31.916470+0000","last_undegraded":"2023-10-17T02:17:31.916470+0000","last_fullsized":"2023-10-17T02:17:31.916470+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'113","last_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_deep_scrub":"95'113","last_deep_scrub_stamp":"2023-10-17T01:48:04.916946+0000","last_clean_scrub_stamp":"2023-10-17T01:48:04.916946+0000","objects_scrubbed":10,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:11.495463+0000","scrub_duration":0.109145037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3139,"num_read_kb":2647,"num_write":53,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1554,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"115'590066","reported_seq":1178845,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:17:32.399291+0000","last_change":"2023-10-17T02:17:21.391206+0000","last_active":"2023-10-17T02:17:32.399291+0000","last_peered":"2023-10-17T02:17:32.399291+0000","last_clean":"2023-10-17T02:17:32.399291+0000","last_became_active":"2023-10-17T00:42:19.130372+0000","last_became_peered":"2023-10-17T00:42:19.130372+0000","last_unstale":"2023-10-17T02:17:32.399291+0000","last_undegraded":"2023-10-17T02:17:32.399291+0000","last_fullsized":"2023-10-17T02:17:32.399291+0000","mapping_epoch":16,"log_start":"115'580018","ondisk_log_start":"115'580018","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'588992","last_scrub_stamp":"2023-10-17T02:17:21.391044+0000","last_deep_scrub":"115'588992","last_deep_scrub_stamp":"2023-10-17T02:17:21.391044+0000","last_clean_scrub_stamp":"2023-10-17T02:17:21.391044+0000","objects_scrubbed":2547,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T10:25:09.951293+0000","scrub_duration":955.97464957900002,"objects_trimmed":0,"snaptrim_duration":0.0022593880000000002,"stat_sum":{"num_bytes":7290507283,"num_objects":2547,"num_object_clones":1273,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":589179,"num_read_kb":2324284,"num_write":587387,"num_write_kb":2349542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"115'11758","reported_seq":16391,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:17:31.903250+0000","last_change":"2023-10-17T01:48:13.741269+0000","last_active":"2023-10-17T02:17:31.903250+0000","last_peered":"2023-10-17T02:17:31.903250+0000","last_clean":"2023-10-17T02:17:31.903250+0000","last_became_active":"2023-10-17T00:42:12.796160+0000","last_became_peered":"2023-10-17T00:42:12.796160+0000","last_unstale":"2023-10-17T02:17:31.903250+0000","last_undegraded":"2023-10-17T02:17:31.903250+0000","last_fullsized":"2023-10-17T02:17:31.903250+0000","mapping_epoch":13,"log_start":"58'1700","ondisk_log_start":"58'1700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'8462","last_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_deep_scrub":"95'8462","last_deep_scrub_stamp":"2023-10-17T01:48:13.741067+0000","last_clean_scrub_stamp":"2023-10-17T01:48:13.741067+0000","objects_scrubbed":9,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:14.091565+0000","scrub_duration":0.22939052300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3150,"num_read_kb":2662,"num_write":50,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1553,"num_omap_keys":33,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"95'99","reported_seq":299,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-10-17T02:14:46.423633+0000","last_change":"2023-10-17T01:48:06.751833+0000","last_active":"2023-10-17T02:14:46.423633+0000","last_peered":"2023-10-17T02:14:46.423633+0000","last_clean":"2023-10-17T02:14:46.423633+0000","last_became_active":"2023-10-17T00:42:10.041745+0000","last_became_peered":"2023-10-17T00:42:10.041745+0000","last_unstale":"2023-10-17T02:14:46.423633+0000","last_undegraded":"2023-10-17T02:14:46.423633+0000","last_fullsized":"2023-10-17T02:14:46.423633+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'99","last_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_deep_scrub":"95'99","last_deep_scrub_stamp":"2023-10-17T01:48:06.751506+0000","last_clean_scrub_stamp":"2023-10-17T01:48:06.751506+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:00.141703+0000","scrub_duration":0.32012012000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29323993107,"num_objects":10241,"num_object_clones":5120,"num_object_copies":20482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10241,"num_whiteouts":0,"num_read":2366483,"num_read_kb":9337480,"num_write":2363637,"num_write_kb":9454542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19365335040,"data_stored":22766194726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40205,"ondisk_log_size":40205,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2341,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":23856,"num_read_kb":20562,"num_write":493,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13660,"num_omap_keys":323,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34223,"internal_metadata":0},"log_size":48401,"ondisk_log_size":48401,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7578,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739516,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11611248,"kb_used_data":9457576,"kb_used_omap":854,"kb_used_meta":2152809,"kb_avail":82760592,"statfs":{"total":96636764160,"available":84746846208,"internally_reserved":0,"allocated":9684557824,"data_stored":11384337226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":875147,"internal_metadata":2204476789},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,4,5,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:16:43 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.46200000000000002,"15min":0.49099999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.158},"max":{"1min":5.7220000000000004,"5min":5.7220000000000004,"15min":12.852},"last":1.399},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.44600000000000001,"15min":0.48699999999999999},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.192},"max":{"1min":5.7560000000000002,"5min":5.7560000000000002,"15min":12.784000000000001},"last":0.877}]},{"osd":1,"last update":"Tue Oct 17 02:17:20 2023","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.41999999999999998,"15min":0.46500000000000002},"min":{"1min":0.22,"5min":0.215,"15min":0.20899999999999999},"max":{"1min":0.42299999999999999,"5min":3.302,"15min":6.202},"last":0.223},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.41499999999999998,"15min":0.56299999999999994},"min":{"1min":0.27600000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.65100000000000002,"5min":2.9380000000000002,"15min":12.824999999999999},"last":0.26300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739512,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5769160,"kb_used_data":4656256,"kb_used_omap":425,"kb_used_meta":1112470,"kb_avail":88602680,"statfs":{"total":96636764160,"available":90729144320,"internally_reserved":0,"allocated":4768006144,"data_stored":5605559145,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":435759,"internal_metadata":1139169745},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,5,9,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 17 02:16:42 2023","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.51600000000000001,"15min":0.628},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.16900000000000001},"max":{"1min":0.54200000000000004,"5min":5.1109999999999998,"15min":11.891999999999999},"last":0.16900000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.61699999999999999,"15min":0.69199999999999995},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.224},"max":{"1min":5.0350000000000001,"5min":5.1449999999999996,"15min":11.926},"last":0.20899999999999999}]},{"osd":2,"last update":"Tue Oct 17 02:16:48 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.64600000000000002,"15min":0.71699999999999997},"min":{"1min":0.20899999999999999,"5min":0.20599999999999999,"15min":0.19500000000000001},"max":{"1min":0.96599999999999997,"5min":13.300000000000001,"15min":25.128},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.50700000000000001,"15min":0.47599999999999998},"min":{"1min":0.26100000000000001,"5min":0.23000000000000001,"15min":0.214},"max":{"1min":4.9980000000000002,"5min":4.9980000000000002,"15min":5.2709999999999999},"last":0.254}]}]},{"osd":0,"up_from":8,"seq":34359739512,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6515372,"kb_used_data":4800100,"kb_used_omap":454,"kb_used_meta":1714809,"kb_avail":87856468,"statfs":{"total":96636764160,"available":89965023232,"internally_reserved":0,"allocated":4915302400,"data_stored":5777700912,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":465730,"internal_metadata":1755964606},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,5,3,15,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 17 02:17:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.68300000000000005,"15min":0.61799999999999999},"min":{"1min":0.192,"5min":0.161,"15min":0.161},"max":{"1min":0.92600000000000005,"5min":6.0389999999999997,"15min":13.834},"last":0.248},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.43099999999999999,"15min":0.56799999999999995},"min":{"1min":0.23200000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.88700000000000001,"5min":4.6900000000000004,"15min":9.0730000000000004},"last":0.35499999999999998}]},{"osd":2,"last update":"Tue Oct 17 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.53600000000000003,"15min":0.60899999999999999},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.2300000000000004,"5min":5.2300000000000004,"15min":12.382},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.752,"5min":0.49299999999999999,"15min":0.60299999999999998},"min":{"1min":0.23100000000000001,"5min":0.221,"15min":0.20200000000000001},"max":{"1min":5.3529999999999998,"5min":5.3529999999999998,"15min":12.561},"last":0.39400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3789,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12301,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14738,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7184,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4914515968,"data_stored":5777113088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4766937088,"data_stored":5604769811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9683881984,"data_stored":11384311827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T02:17:34.967 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-17T02:17:35.660 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-17T02:17:35.661 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-17T02:17:35.662 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-17T02:17:35.662 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T02:17:36.539 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-10-17T02:17:41.764 INFO:tasks.ceph.osd.0:Stopped 2023-10-17T02:17:41.765 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-17T02:17:41.765 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T02:17:41.941 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~5s 2023-10-17T02:17:47.244 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-10-17T02:17:47.245 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-10-17T02:17:47.867 INFO:tasks.ceph.osd.1:Stopped 2023-10-17T02:17:47.868 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-17T02:17:47.869 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T02:17:52.447 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-10-17T02:17:52.447 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-10-17T02:17:53.970 INFO:tasks.ceph.osd.2:Stopped 2023-10-17T02:17:53.971 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-17T02:17:53.971 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-17T02:17:53.972 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T02:17:54.003 INFO:tasks.ceph.mgr.x:Stopped 2023-10-17T02:17:54.004 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-17T02:17:54.004 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-17T02:17:54.005 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T02:17:54.072 INFO:tasks.ceph.mon.a:Stopped 2023-10-17T02:17:54.073 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-17T02:17:54.073 DEBUG:teuthology.orchestra.run.smithi169:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-10-17T02:17:54.215 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi169.front.sepia.ceph.com 2023-10-17T02:17:54.215 DEBUG:teuthology.orchestra.run.smithi169:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-17T02:17:56.563 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi169.front.sepia.ceph.com 2023-10-17T02:17:56.564 DEBUG:teuthology.orchestra.run.smithi169:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-17T02:17:57.005 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi169.front.sepia.ceph.com 2023-10-17T02:17:57.005 DEBUG:teuthology.orchestra.run.smithi169:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-17T02:17:57.448 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-10-17T02:17:57.624 INFO:tasks.ceph:Archiving mon data... 2023-10-17T02:17:57.819 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rraja-2023-10-17_00:10:32-rbd:cli-debug-62994-distro-default-smithi/7430972/data/mon.a.tgz 2023-10-17T02:17:57.819 DEBUG:teuthology.orchestra.run.smithi169:> mktemp 2023-10-17T02:17:57.826 INFO:teuthology.orchestra.run.smithi169.stdout:/tmp/tmp.Ow4mp9eVga 2023-10-17T02:17:57.826 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.Ow4mp9eVga 2023-10-17T02:17:57.998 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0666 /tmp/tmp.Ow4mp9eVga 2023-10-17T02:17:58.096 DEBUG:teuthology.orchestra.remote:smithi169:/tmp/tmp.Ow4mp9eVga is 502KB 2023-10-17T02:17:58.167 DEBUG:teuthology.orchestra.run.smithi169:> rm -fr /tmp/tmp.Ow4mp9eVga 2023-10-17T02:17:58.174 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-17T02:17:58.174 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-17T02:17:58.259 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-17T02:17:58.260 DEBUG:teuthology.orchestra.run.smithi169:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-17T02:17:58.331 INFO:teuthology.orchestra.run.smithi169.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-17T02:17:58.339 INFO:tasks.ceph:Archiving crash dumps... 2023-10-17T02:17:58.343 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/lib/ceph/crash to /home/teuthworker/archive/rraja-2023-10-17_00:10:32-rbd:cli-debug-62994-distro-default-smithi/7430972/remote/smithi169/crash 2023-10-17T02:17:58.349 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-17T02:17:58.399 INFO:tasks.ceph:Compressing logs... 2023-10-17T02:17:58.400 DEBUG:teuthology.orchestra.run.smithi169:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-17T02:17:58.477 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54659.log 2023-10-17T02:17:58.479 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45452.log 2023-10-17T02:17:58.480 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.54659.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54659.log.gz 2023-10-17T02:17:58.480 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37636.log 2023-10-17T02:17:58.480 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.45452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51904.log 2023-10-17T02:17:58.480 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.37636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41645.log 2023-10-17T02:17:58.481 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38650.log 2023-10-17T02:17:58.486 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43268.log 2023-10-17T02:17:58.486 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.41645.log: /var/log/ceph/ceph-client.admin.38650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40711.log 2023-10-17T02:17:58.486 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.43268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50939.log 2023-10-17T02:17:58.487 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51904.log.gz 2023-10-17T02:17:58.487 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.40711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51670.log 2023-10-17T02:17:58.487 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.50939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51517.log 2023-10-17T02:17:58.487 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50939.log.gz 2023-10-17T02:17:58.488 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48284.log 2023-10-17T02:17:58.488 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51670.log.gz 2023-10-17T02:17:58.488 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52270.log 2023-10-17T02:17:58.489 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51517.log.gz 2023-10-17T02:17:58.489 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.48284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38866.log 2023-10-17T02:17:58.489 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48284.log.gz 2023-10-17T02:17:58.489 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38633.log 2023-10-17T02:17:58.490 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52270.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52270.log.gz 2023-10-17T02:17:58.490 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42063.log 2023-10-17T02:17:58.490 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41048.log 2023-10-17T02:17:58.491 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.42063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54823.log 2023-10-17T02:17:58.491 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.41048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47435.log 2023-10-17T02:17:58.491 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.54823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39460.log 2023-10-17T02:17:58.491 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54823.log.gz 2023-10-17T02:17:58.492 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.47435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53008.log 2023-10-17T02:17:58.492 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47435.log.gz 2023-10-17T02:17:58.492 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38616.log 2023-10-17T02:17:58.493 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49049.log 2023-10-17T02:17:58.493 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53008.log.gz 2023-10-17T02:17:58.493 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39156.log 2023-10-17T02:17:58.677 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.49049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40988.log 2023-10-17T02:17:58.678 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49049.log.gz 2023-10-17T02:17:58.678 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53833.log 2023-10-17T02:17:58.679 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.40988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44339.log 2023-10-17T02:17:58.679 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47097.log 2023-10-17T02:17:58.679 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53833.log.gz 2023-10-17T02:17:58.680 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.44339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38920.log 2023-10-17T02:17:58.680 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.47097.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.47097.log.gz 2023-10-17T02:17:58.680 INFO:teuthology.orchestra.run.smithi169.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.51031.log 2023-10-17T02:17:58.680 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52348.log 2023-10-17T02:17:58.681 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39460.log.gz -- replaced with /var/log/ceph/ceph-client.admin.38866.log.gz 2023-10-17T02:17:58.681 INFO:teuthology.orchestra.run.smithi169.stderr: 2023-10-17T02:17:58.681 INFO:teuthology.orchestra.run.smithi169.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39156.log.gz 2023-10-17T02:17:58.681 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38616.log.gz 0.0% 2023-10-17T02:17:58.682 INFO:teuthology.orchestra.run.smithi169.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41645.log.gz 2023-10-17T02:17:58.682 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43268.log.gz -- replaced with /var/log/ceph/ceph-client.admin.41048.log.gz 2023-10-17T02:17:58.682 INFO:teuthology.orchestra.run.smithi169.stderr: 2023-10-17T02:17:58.682 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45452.log.gz 2023-10-17T02:17:58.683 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38633.log.gz 2023-10-17T02:17:58.683 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38650.log.gz 2023-10-17T02:17:58.683 INFO:teuthology.orchestra.run.smithi169.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40711.log.gz 2023-10-17T02:17:58.684 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42063.log.gz 2023-10-17T02:17:58.684 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40988.log.gz 2023-10-17T02:17:58.684 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44339.log.gz 2023-10-17T02:17:58.684 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38920.log.gz 2023-10-17T02:17:58.685 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51062.log 2023-10-17T02:17:58.685 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51031.log.gz 2023-10-17T02:17:58.685 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52348.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.52348.log.gz -5 2023-10-17T02:17:58.685 INFO:teuthology.orchestra.run.smithi169.stderr: --verbose -- /var/log/ceph/ceph-client.admin.51001.log 2023-10-17T02:17:58.686 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54311.log 2023-10-17T02:17:58.686 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51062.log.gz 2023-10-17T02:17:58.686 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41419.log 2023-10-17T02:17:58.687 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51001.log.gz 2023-10-17T02:17:58.687 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.54311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45510.log 2023-10-17T02:17:58.687 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54311.log.gz 2023-10-17T02:17:58.687 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43700.log 2023-10-17T02:17:58.688 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.41419.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41419.log.gz 2023-10-17T02:17:58.688 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.45510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38431.log 2023-10-17T02:17:58.688 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45510.log.gz 2023-10-17T02:17:58.688 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.43700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54445.log 2023-10-17T02:17:58.689 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43700.log.gz 2023-10-17T02:17:58.689 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52560.log 2023-10-17T02:17:58.689 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38431.log.gz 2023-10-17T02:17:58.690 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.54445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48376.log 2023-10-17T02:17:58.690 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54445.log.gz 2023-10-17T02:17:58.690 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42408.log 2023-10-17T02:17:58.690 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52560.log.gz 2023-10-17T02:17:58.691 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.48376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50116.log 2023-10-17T02:17:58.691 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48376.log.gz 2023-10-17T02:17:58.691 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.42408.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37636.log.gz 2023-10-17T02:17:58.691 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45428.log 2023-10-17T02:17:58.692 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42408.log.gz 2023-10-17T02:17:58.692 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.50116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54472.log 2023-10-17T02:17:58.692 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50116.log.gz 2023-10-17T02:17:58.693 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.45428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44383.log 2023-10-17T02:17:58.693 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45428.log.gz 2023-10-17T02:17:58.693 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.54472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38687.log 2023-10-17T02:17:58.693 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54472.log.gz 2023-10-17T02:17:58.694 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.44383.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.41322.log -- replaced with /var/log/ceph/ceph-client.admin.44383.log.gz 2023-10-17T02:17:58.694 INFO:teuthology.orchestra.run.smithi169.stderr: 2023-10-17T02:17:58.694 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40742.log 2023-10-17T02:17:58.694 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38687.log.gz 2023-10-17T02:17:58.695 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.41322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50086.log 2023-10-17T02:17:58.695 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41322.log.gz 2023-10-17T02:17:58.695 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.40742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44124.log 2023-10-17T02:17:58.696 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40742.log.gz 2023-10-17T02:17:58.696 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.50086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52400.log 2023-10-17T02:17:58.696 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50086.log.gz 2023-10-17T02:17:58.696 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.44124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55090.log 2023-10-17T02:17:58.697 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44124.log.gz 2023-10-17T02:17:58.697 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52400.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.52400.log.gz 2023-10-17T02:17:58.697 INFO:teuthology.orchestra.run.smithi169.stderr: -- /var/log/ceph/ceph-client.admin.53514.log 2023-10-17T02:17:58.697 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.55090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48685.log 2023-10-17T02:17:58.698 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55090.log.gz 2023-10-17T02:17:58.698 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39244.log 2023-10-17T02:17:58.698 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53514.log.gz 2023-10-17T02:17:58.699 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.48685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50817.log 2023-10-17T02:17:58.699 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48685.log.gz 2023-10-17T02:17:58.699 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38957.log 2023-10-17T02:17:58.699 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39244.log.gz 2023-10-17T02:17:58.700 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5/var/log/ceph/ceph-client.admin.50817.log: --verbose -- /var/log/ceph/ceph-client.admin.49511.log 2023-10-17T02:17:58.700 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50817.log.gz 2023-10-17T02:17:58.700 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41174.log 2023-10-17T02:17:58.700 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38957.log.gz 2023-10-17T02:17:58.701 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.49511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47466.log 2023-10-17T02:17:58.701 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49511.log.gz 2023-10-17T02:17:58.701 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.41174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37372.log 2023-10-17T02:17:58.702 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41174.log.gz 2023-10-17T02:17:58.702 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.47466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34397.log 2023-10-17T02:17:58.703 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47466.log.gz 2023-10-17T02:17:58.703 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.37372.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-17T02:17:58.704 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37372.log.gz 2023-10-17T02:17:58.704 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.34397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43076.log 2023-10-17T02:17:58.705 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34397.log.gz 2023-10-17T02:17:58.705 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44866.log 2023-10-17T02:17:58.706 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.43076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51336.log 2023-10-17T02:17:58.706 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43076.log.gz 2023-10-17T02:17:58.707 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.44866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52296.log 2023-10-17T02:17:58.707 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44866.log.gz 2023-10-17T02:17:58.708 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39048.log 2023-10-17T02:17:58.708 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51336.log.gz 2023-10-17T02:17:58.709 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46550.log 2023-10-17T02:17:58.709 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52296.log.gz 2023-10-17T02:17:58.710 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52824.log 2023-10-17T02:17:58.710 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39048.log.gz 2023-10-17T02:17:58.711 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.46550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47647.log 2023-10-17T02:17:58.711 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46550.log.gz 2023-10-17T02:17:58.711 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52374.log 2023-10-17T02:17:58.712 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52824.log.gz 2023-10-17T02:17:58.712 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.47647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46731.log 2023-10-17T02:17:58.712 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47647.log.gz 2023-10-17T02:17:58.712 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38508.log 2023-10-17T02:17:58.713 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52374.log.gz 2023-10-17T02:17:58.713 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.46731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39956.log 2023-10-17T02:17:58.713 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46731.log.gz 2023-10-17T02:17:58.713 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52638.log 2023-10-17T02:17:58.714 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38508.log.gz 2023-10-17T02:17:58.714 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42522.log 2023-10-17T02:17:58.714 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39956.log.gz 2023-10-17T02:17:58.715 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37706.log 2023-10-17T02:17:58.715 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52638.log.gz 2023-10-17T02:17:58.715 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.42522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44197.log 2023-10-17T02:17:58.715 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42522.log.gz 2023-10-17T02:17:58.716 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.37706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51639.log 2023-10-17T02:17:58.716 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37706.log.gz 2023-10-17T02:17:58.716 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.44197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49904.log 2023-10-17T02:17:58.717 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44197.log.gz 2023-10-17T02:17:58.717 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45400.log 2023-10-17T02:17:58.718 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51639.log.gz 2023-10-17T02:17:58.718 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.49904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52052.log 2023-10-17T02:17:58.719 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49904.log.gz 2023-10-17T02:17:58.719 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.45400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38741.log 2023-10-17T02:17:58.719 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45400.log.gz 2023-10-17T02:17:58.719 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39102.log 2023-10-17T02:17:58.720 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52052.log.gz 2023-10-17T02:17:58.720 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51213.log 2023-10-17T02:17:58.720 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38741.log.gz 2023-10-17T02:17:58.721 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42432.log 2023-10-17T02:17:58.721 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39102.log.gz 2023-10-17T02:17:58.721 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50241.log 2023-10-17T02:17:58.721 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51213.log.gz 2023-10-17T02:17:58.722 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.42432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47253.log 2023-10-17T02:17:58.722 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42432.log.gz 2023-10-17T02:17:58.722 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.50241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53912.log 2023-10-17T02:17:58.723 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50241.log.gz 2023-10-17T02:17:58.723 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.47253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42276.log 2023-10-17T02:17:58.723 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47253.log.gz 2023-10-17T02:17:58.724 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49170.log 2023-10-17T02:17:58.724 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53912.log.gz 2023-10-17T02:17:58.724 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.42276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54204.log 2023-10-17T02:17:58.724 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42276.log.gz 2023-10-17T02:17:58.725 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.49170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41262.log 2023-10-17T02:17:58.725 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49170.log.gz 2023-10-17T02:17:58.725 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.54204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46701.log 2023-10-17T02:17:58.726 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54204.log.gz 2023-10-17T02:17:58.726 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.41262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45673.log 2023-10-17T02:17:58.727 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41262.log.gz 2023-10-17T02:17:58.727 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.46701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54929.log 2023-10-17T02:17:58.727 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46701.log.gz 2023-10-17T02:17:58.728 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.45673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52716.log 2023-10-17T02:17:58.728 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45673.log.gz 2023-10-17T02:17:58.728 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.54929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55332.log 2023-10-17T02:17:58.729 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54929.log.gz 2023-10-17T02:17:58.729 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53436.log 2023-10-17T02:17:58.729 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52716.log.gz 2023-10-17T02:17:58.730 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.55332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49388.log 2023-10-17T02:17:58.730 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55332.log.gz 2023-10-17T02:17:58.730 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48563.log 2023-10-17T02:17:58.730 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53436.log.gz 2023-10-17T02:17:58.731 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.49388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53245.log 2023-10-17T02:17:58.731 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49388.log.gz 2023-10-17T02:17:58.731 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.48563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45601.log 2023-10-17T02:17:58.731 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48563.log.gz 2023-10-17T02:17:58.732 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54717.log 2023-10-17T02:17:58.732 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53245.log.gz 2023-10-17T02:17:58.732 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.45601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40958.log 2023-10-17T02:17:58.733 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45601.log.gz 2023-10-17T02:17:58.733 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.54717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47739.log 2023-10-17T02:17:58.733 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54717.log.gz 2023-10-17T02:17:58.733 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.40958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42490.log 2023-10-17T02:17:58.734 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40958.log.gz 2023-10-17T02:17:58.734 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.47739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39065.log 2023-10-17T02:17:58.734 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47739.log.gz 2023-10-17T02:17:58.735 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.42490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41078.log 2023-10-17T02:17:58.735 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42490.log.gz 2023-10-17T02:17:58.735 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52015.log 2023-10-17T02:17:58.735 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39065.log.gz 2023-10-17T02:17:58.736 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.41078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37446.log 2023-10-17T02:17:58.736 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41078.log.gz 2023-10-17T02:17:58.736 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49231.log 2023-10-17T02:17:58.736 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52015.log.gz 2023-10-17T02:17:58.737 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.37446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39788.log 2023-10-17T02:17:58.737 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37446.log.gz 2023-10-17T02:17:58.737 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.49231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47677.log 2023-10-17T02:17:58.738 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49231.log.gz 2023-10-17T02:17:58.738 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39872.log 2023-10-17T02:17:58.738 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39788.log.gz 2023-10-17T02:17:58.738 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.47677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43015.log 2023-10-17T02:17:58.739 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47677.log.gz 2023-10-17T02:17:58.739 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53275.log 2023-10-17T02:17:58.739 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39872.log.gz 2023-10-17T02:17:58.739 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.43015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52930.log 2023-10-17T02:17:58.740 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43015.log.gz 2023-10-17T02:17:58.740 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50725.log 2023-10-17T02:17:58.740 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53275.log.gz 2023-10-17T02:17:58.740 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50178.log 2023-10-17T02:17:58.741 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52930.log.gz 2023-10-17T02:17:58.741 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.50725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37472.log 2023-10-17T02:17:58.741 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50725.log.gz 2023-10-17T02:17:58.741 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.50178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51396.log 2023-10-17T02:17:58.742 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50178.log.gz 2023-10-17T02:17:58.742 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.37472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44833.log 2023-10-17T02:17:58.742 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37472.log.gz 2023-10-17T02:17:58.743 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39281.log 2023-10-17T02:17:58.743 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51396.log.gz 2023-10-17T02:17:58.743 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.44833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48926.log 2023-10-17T02:17:58.743 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44833.log.gz 2023-10-17T02:17:58.744 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55384.log 2023-10-17T02:17:58.744 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39281.log.gz 2023-10-17T02:17:58.744 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.48926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40832.log 2023-10-17T02:17:58.744 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48926.log.gz 2023-10-17T02:17:58.745 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.55384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41513.log 2023-10-17T02:17:58.745 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55384.log.gz 2023-10-17T02:17:58.745 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.40832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52769.log 2023-10-17T02:17:58.745 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40832.log.gz 2023-10-17T02:17:58.746 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.41513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44990.log 2023-10-17T02:17:58.746 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41513.log.gz 2023-10-17T02:17:58.746 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37524.log 2023-10-17T02:17:58.746 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52769.log.gz 2023-10-17T02:17:58.747 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.44990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46488.log 2023-10-17T02:17:58.747 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44990.log.gz 2023-10-17T02:17:58.747 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.37524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47951.log 2023-10-17T02:17:58.748 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37524.log.gz 2023-10-17T02:17:58.748 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.46488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34491.log 2023-10-17T02:17:58.748 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46488.log.gz 2023-10-17T02:17:58.748 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.47951.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-17T02:17:58.749 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47951.log.gz 2023-10-17T02:17:58.749 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.34491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38414.log 2023-10-17T02:17:58.749 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34491.log.gz 2023-10-17T02:17:58.749 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38579.log 2023-10-17T02:17:58.750 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46761.log 2023-10-17T02:17:58.750 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38414.log.gz 2023-10-17T02:17:58.750 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52089.log 2023-10-17T02:17:58.751 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38579.log.gz 2023-10-17T02:17:58.751 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.46761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37498.log 2023-10-17T02:17:58.751 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46761.log.gz 2023-10-17T02:17:58.752 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48625.log 2023-10-17T02:17:58.752 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52089.log.gz 2023-10-17T02:17:58.752 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.37498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38525.log 2023-10-17T02:17:58.752 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37498.log.gz 2023-10-17T02:17:58.753 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.48625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44740.log 2023-10-17T02:17:58.753 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48625.log.gz 2023-10-17T02:17:58.753 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42137.log 2023-10-17T02:17:58.753 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38525.log.gz 2023-10-17T02:17:58.754 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.44740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55011.log 2023-10-17T02:17:58.754 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44740.log.gz 2023-10-17T02:17:58.754 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.42137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34636.log 2023-10-17T02:17:58.755 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42137.log.gz 2023-10-17T02:17:58.755 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.55011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41941.log 2023-10-17T02:17:58.755 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55011.log.gz 2023-10-17T02:17:58.755 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.34636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55064.log 2023-10-17T02:17:58.756 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34636.log.gz 2023-10-17T02:17:58.756 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.41941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49572.log 2023-10-17T02:17:58.756 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41941.log.gz 2023-10-17T02:17:58.757 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.55064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48315.log 2023-10-17T02:17:58.757 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55064.log.gz 2023-10-17T02:17:58.757 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.49572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39660.log 2023-10-17T02:17:58.757 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49572.log.gz 2023-10-17T02:17:58.758 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.48315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50211.log 2023-10-17T02:17:58.758 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48315.log.gz 2023-10-17T02:17:58.758 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50695.log 2023-10-17T02:17:58.759 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39660.log.gz 2023-10-17T02:17:58.759 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.50211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46427.log 2023-10-17T02:17:58.759 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50211.log.gz 2023-10-17T02:17:58.759 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.50695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40602.log 2023-10-17T02:17:58.760 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50695.log.gz 2023-10-17T02:17:58.760 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.46427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53301.log 2023-10-17T02:17:58.760 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46427.log.gz 2023-10-17T02:17:58.760 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.40602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38903.log 2023-10-17T02:17:58.761 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40602.log.gz 2023-10-17T02:17:58.761 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53301.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53301.log.gz 2023-10-17T02:17:58.761 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37716.log 2023-10-17T02:17:58.761 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-17T02:17:58.762 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38903.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38903.log.gz 2023-10-17T02:17:58.762 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.37716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52795.log 2023-10-17T02:17:58.762 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37716.log.gz 2023-10-17T02:17:58.762 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44710.log 2023-10-17T02:17:58.763 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42708.log 2023-10-17T02:17:58.763 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52795.log.gz 2023-10-17T02:17:58.763 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.44710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54850.log 2023-10-17T02:17:58.764 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.42708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49140.log 2023-10-17T02:17:58.764 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42708.log.gz 2023-10-17T02:17:58.764 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.54850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45235.log 2023-10-17T02:17:58.764 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54850.log.gz 2023-10-17T02:17:58.765 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.49140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46821.log 2023-10-17T02:17:58.765 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49140.log.gz 2023-10-17T02:17:58.765 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.45235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51976.log 2023-10-17T02:17:58.766 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45235.log.gz 2023-10-17T02:17:58.766 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40907.log/var/log/ceph/ceph-client.admin.46821.log: 2023-10-17T02:17:58.766 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46821.log.gz 2023-10-17T02:17:58.766 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34574.log 2023-10-17T02:17:58.767 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51976.log.gz 2023-10-17T02:17:58.767 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.40907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43159.log 2023-10-17T02:17:58.767 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40907.log.gz 2023-10-17T02:17:58.768 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49602.log 2023-10-17T02:17:58.768 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.34574.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34574.log.gz 2023-10-17T02:17:58.768 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.43159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38849.log 2023-10-17T02:17:58.769 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43159.log.gz 2023-10-17T02:17:58.769 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.49602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50363.log 2023-10-17T02:17:58.769 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49602.log.gz 2023-10-17T02:17:58.769 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37836.log 2023-10-17T02:17:58.770 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38849.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38849.log.gz 2023-10-17T02:17:58.770 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.50363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50271.log 2023-10-17T02:17:58.770 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50363.log.gz 2023-10-17T02:17:58.771 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.37836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47404.log 2023-10-17T02:17:58.771 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37836.log.gz 2023-10-17T02:17:58.771 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55305.log 2023-10-17T02:17:58.771 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.50271.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50271.log.gz 2023-10-17T02:17:58.772 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.47404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41573.log 2023-10-17T02:17:58.772 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47404.log.gz 2023-10-17T02:17:58.772 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.55305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47829.log 2023-10-17T02:17:58.773 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55305.log.gz 2023-10-17T02:17:58.773 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38724.log 2023-10-17T02:17:58.774 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.41573.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41573.log.gz 2023-10-17T02:17:58.774 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.47829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37556.log 2023-10-17T02:17:58.774 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47829.log.gz 2023-10-17T02:17:58.775 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45479.log 2023-10-17T02:17:58.775 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38724.log.gz 2023-10-17T02:17:58.775 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.37556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48498.log 2023-10-17T02:17:58.775 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37556.log.gz 2023-10-17T02:17:58.776 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.45479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52612.log 2023-10-17T02:17:58.776 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45479.log.gz 2023-10-17T02:17:58.777 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.48498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43107.log 2023-10-17T02:17:58.777 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48498.log.gz 2023-10-17T02:17:58.777 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53141.log 2023-10-17T02:17:58.778 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52612.log.gz 2023-10-17T02:17:58.778 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.43107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48806.log 2023-10-17T02:17:58.778 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43107.log.gz 2023-10-17T02:17:58.778 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52244.log 2023-10-17T02:17:58.779 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53141.log.gz 2023-10-17T02:17:58.779 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53089.log 2023-10-17T02:17:58.779 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.48806.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48806.log.gz 2023-10-17T02:17:58.780 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55224.log 2023-10-17T02:17:58.780 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52244.log.gz 2023-10-17T02:17:58.780 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51886.log 2023-10-17T02:17:58.781 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53089.log.gz 2023-10-17T02:17:58.781 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44771.log 2023-10-17T02:17:58.781 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.55224.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55224.log.gz 2023-10-17T02:17:58.781 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42629.log 2023-10-17T02:17:58.782 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51886.log.gz 2023-10-17T02:17:58.782 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.44771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54552.log 2023-10-17T02:17:58.782 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44771.log.gz 2023-10-17T02:17:58.782 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49965.log 2023-10-17T02:17:58.783 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.42629.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42629.log.gz 2023-10-17T02:17:58.783 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.54552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41018.log 2023-10-17T02:17:58.784 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54552.log.gz 2023-10-17T02:17:58.784 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.49965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51182.log 2023-10-17T02:17:58.784 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49965.log.gz 2023-10-17T02:17:58.784 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40259.log 2023-10-17T02:17:58.785 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.41018.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41018.log.gz 2023-10-17T02:17:58.785 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47707.log 2023-10-17T02:17:58.785 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51182.log.gz 2023-10-17T02:17:58.785 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.40259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50025.log 2023-10-17T02:17:58.786 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40259.log.gz 2023-10-17T02:17:58.786 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46855.log 2023-10-17T02:17:58.786 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.47707.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47707.log.gz 2023-10-17T02:17:58.787 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.50025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39352.log 2023-10-17T02:17:58.787 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50025.log.gz 2023-10-17T02:17:58.787 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.46855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43648.log 2023-10-17T02:17:58.787 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46855.log.gz 2023-10-17T02:17:58.788 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44309.log 2023-10-17T02:17:58.788 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39352.log.gz 2023-10-17T02:17:58.788 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.43648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43894.log 2023-10-17T02:17:58.789 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43648.log.gz 2023-10-17T02:17:58.789 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.44309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51958.log 2023-10-17T02:17:58.789 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44309.log.gz 2023-10-17T02:17:58.789 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.43894.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43894.log.gz -5 2023-10-17T02:17:58.790 INFO:teuthology.orchestra.run.smithi169.stderr: --verbose -- /var/log/ceph/ceph-client.admin.50545.log 2023-10-17T02:17:58.790 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51958.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51958.log.gz 2023-10-17T02:17:58.790 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38471.log 2023-10-17T02:17:58.791 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37710.log 2023-10-17T02:17:58.791 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.50545.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50545.log.gz 2023-10-17T02:17:58.791 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54337.log 2023-10-17T02:17:58.791 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38471.log.gz 2023-10-17T02:17:58.792 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.37710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40292.log 2023-10-17T02:17:58.792 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37710.log.gz 2023-10-17T02:17:58.792 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.54337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45020.log 2023-10-17T02:17:58.792 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54337.log.gz 2023-10-17T02:17:58.793 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.40292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39372.log 2023-10-17T02:17:58.793 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40292.log.gz 2023-10-17T02:17:58.794 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.45020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47799.log 2023-10-17T02:17:58.794 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45020.log.gz 2023-10-17T02:17:58.794 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54771.log 2023-10-17T02:17:58.794 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39372.log.gz 2023-10-17T02:17:58.795 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.47799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39426.log 2023-10-17T02:17:58.795 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47799.log.gz 2023-10-17T02:17:58.795 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.54771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49358.log 2023-10-17T02:17:58.795 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54771.log.gz 2023-10-17T02:17:58.796 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53488.log 2023-10-17T02:17:58.796 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39426.log.gz 2023-10-17T02:17:58.796 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.49358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41972.log 2023-10-17T02:17:58.796 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49358.log.gz 2023-10-17T02:17:58.797 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55279.log 2023-10-17T02:17:58.797 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53488.log.gz 2023-10-17T02:17:58.797 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.41972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43729.log 2023-10-17T02:17:58.798 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41972.log.gz 2023-10-17T02:17:58.798 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.55279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43676.log 2023-10-17T02:17:58.798 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55279.log.gz 2023-10-17T02:17:58.798 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.43729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46641.log 2023-10-17T02:17:58.799 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43729.log.gz 2023-10-17T02:17:58.799 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.43676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42380.log 2023-10-17T02:17:58.799 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43676.log.gz 2023-10-17T02:17:58.799 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.46641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51244.log 2023-10-17T02:17:58.800 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46641.log.gz 2023-10-17T02:17:58.800 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.42380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42905.log 2023-10-17T02:17:58.800 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42380.log.gz 2023-10-17T02:17:58.800 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42246.log 2023-10-17T02:17:58.801 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51244.log.gz 2023-10-17T02:17:58.801 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.42905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54177.log 2023-10-17T02:17:58.801 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42905.log.gz 2023-10-17T02:17:58.801 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.42246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39746.log 2023-10-17T02:17:58.802 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42246.log.gz 2023-10-17T02:17:58.802 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.54177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48715.log 2023-10-17T02:17:58.802 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54177.log.gz 2023-10-17T02:17:58.803 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52125.log 2023-10-17T02:17:58.803 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39746.log.gz 2023-10-17T02:17:58.803 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.48715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54500.log 2023-10-17T02:17:58.804 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48715.log.gz 2023-10-17T02:17:58.804 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52851.log 2023-10-17T02:17:58.804 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52125.log.gz 2023-10-17T02:17:58.805 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.54500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54283.log 2023-10-17T02:17:58.805 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54500.log.gz 2023-10-17T02:17:58.805 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49079.log 2023-10-17T02:17:58.806 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52851.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52851.log.gz 2023-10-17T02:17:58.806 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.54283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47283.log 2023-10-17T02:17:58.806 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54283.log.gz 2023-10-17T02:17:58.807 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.49079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53596.log 2023-10-17T02:17:58.807 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49079.log.gz 2023-10-17T02:17:58.807 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.47283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48866.log 2023-10-17T02:17:58.807 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47283.log.gz 2023-10-17T02:17:58.808 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51831.log 2023-10-17T02:17:58.808 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53596.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53596.log.gz 2023-10-17T02:17:58.808 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.48866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52192.log 2023-10-17T02:17:58.808 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48866.log.gz 2023-10-17T02:17:58.809 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51831.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51831.log.gz 2023-10-17T02:17:58.809 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49448.log 2023-10-17T02:17:58.809 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38380.log 2023-10-17T02:17:58.809 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52192.log.gz 2023-10-17T02:17:58.810 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.49448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43403.log 2023-10-17T02:17:58.810 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49448.log.gz 2023-10-17T02:17:58.810 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41231.log 2023-10-17T02:17:58.810 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38380.log.gz 2023-10-17T02:17:58.811 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.43403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53939.log 2023-10-17T02:17:58.811 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43403.log.gz 2023-10-17T02:17:58.811 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.41231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39190.log 2023-10-17T02:17:58.811 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41231.log.gz 2023-10-17T02:17:58.812 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48103.log 2023-10-17T02:17:58.812 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53939.log.gz 2023-10-17T02:17:58.812 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40384.log 2023-10-17T02:17:58.812 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39190.log.gz 2023-10-17T02:17:58.813 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.48103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40555.log 2023-10-17T02:17:58.813 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48103.log.gz 2023-10-17T02:17:58.813 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.40384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50787.log 2023-10-17T02:17:58.813 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40384.log.gz 2023-10-17T02:17:58.814 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.40555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51849.log 2023-10-17T02:17:58.814 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40555.log.gz 2023-10-17T02:17:58.815 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.50787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42584.log 2023-10-17T02:17:58.815 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50787.log.gz 2023-10-17T02:17:58.815 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37861.log 2023-10-17T02:17:58.815 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51849.log.gz 2023-10-17T02:17:58.816 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.42584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51487.log 2023-10-17T02:17:58.816 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42584.log.gz 2023-10-17T02:17:58.816 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.37861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38795.log 2023-10-17T02:17:58.816 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37861.log.gz 2023-10-17T02:17:58.817 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44248.log 2023-10-17T02:17:58.817 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51487.log.gz 2023-10-17T02:17:58.817 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45326.log 2023-10-17T02:17:58.817 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38795.log.gz 2023-10-17T02:17:58.818 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.44248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47496.log 2023-10-17T02:17:58.818 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44248.log.gz 2023-10-17T02:17:58.818 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.45326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41483.log 2023-10-17T02:17:58.819 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45326.log.gz 2023-10-17T02:17:58.819 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.47496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53462.log 2023-10-17T02:17:58.819 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47496.log.gz 2023-10-17T02:17:58.819 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.41483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54017.log 2023-10-17T02:17:58.820 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41483.log.gz 2023-10-17T02:17:58.820 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55358.log 2023-10-17T02:17:58.820 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53462.log.gz 2023-10-17T02:17:58.820 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.54017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54043.log 2023-10-17T02:17:58.821 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54017.log.gz 2023-10-17T02:17:58.821 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.55358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54579.log 2023-10-17T02:17:58.821 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55358.log.gz 2023-10-17T02:17:58.822 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.54043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42306.log 2023-10-17T02:17:58.822 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54043.log.gz 2023-10-17T02:17:58.822 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43299.log 2023-10-17T02:17:58.823 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.54579.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54579.log.gz 2023-10-17T02:17:58.823 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.42306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42741.log 2023-10-17T02:17:58.823 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42306.log.gz 2023-10-17T02:17:58.823 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.43299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52218.log 2023-10-17T02:17:58.824 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43299.log.gz 2023-10-17T02:17:58.824 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.42741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53167.log 2023-10-17T02:17:58.824 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42741.log.gz 2023-10-17T02:17:58.825 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51274.log 2023-10-17T02:17:58.825 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52218.log.gz 2023-10-17T02:17:58.826 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45944.log 2023-10-17T02:17:58.826 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53167.log.gz 2023-10-17T02:17:58.826 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38451.log 2023-10-17T02:17:58.827 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51274.log.gz 2023-10-17T02:17:58.827 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.45944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52506.log 2023-10-17T02:17:58.827 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45944.log.gz 2023-10-17T02:17:58.828 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43820.log 2023-10-17T02:17:58.828 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38451.log.gz 2023-10-17T02:17:58.828 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50848.log 2023-10-17T02:17:58.829 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52506.log.gz 2023-10-17T02:17:58.829 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.43820.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.43820.log.gz /var/log/ceph/ceph-client.admin.48655.log 2023-10-17T02:17:58.829 INFO:teuthology.orchestra.run.smithi169.stderr: 2023-10-17T02:17:58.830 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.50848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47769.log 2023-10-17T02:17:58.830 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50848.log.gz 2023-10-17T02:17:58.830 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.48655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54985.log 2023-10-17T02:17:58.830 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48655.log.gz 2023-10-17T02:17:58.831 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.47769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42165.log 2023-10-17T02:17:58.831 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47769.log.gz 2023-10-17T02:17:58.831 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.54985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45697.log 2023-10-17T02:17:58.832 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54985.log.gz 2023-10-17T02:17:58.832 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.42165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49874.log 2023-10-17T02:17:58.832 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42165.log.gz 2023-10-17T02:17:58.832 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.45697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45814.log 2023-10-17T02:17:58.833 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45697.log.gz 2023-10-17T02:17:58.833 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.49874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55142.log 2023-10-17T02:17:58.833 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49874.log.gz 2023-10-17T02:17:58.834 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.45814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49663.log 2023-10-17T02:17:58.834 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45814.log.gz 2023-10-17T02:17:58.834 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.55142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54797.log 2023-10-17T02:17:58.834 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55142.log.gz 2023-10-17T02:17:58.835 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.49663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41150.log 2023-10-17T02:17:58.835 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49663.log.gz 2023-10-17T02:17:58.835 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.54797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42933.log 2023-10-17T02:17:58.836 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54797.log.gz 2023-10-17T02:17:58.836 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.41150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55470.log 2023-10-17T02:17:58.836 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41150.log.gz 2023-10-17T02:17:58.837 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.42933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43211.log 2023-10-17T02:17:58.837 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42933.log.gz 2023-10-17T02:17:58.837 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.55470.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55470.log.gz 2023-10-17T02:17:58.837 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40218.log 2023-10-17T02:17:58.838 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.43211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38542.log 2023-10-17T02:17:58.838 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43211.log.gz 2023-10-17T02:17:58.838 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.40218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52143.log 2023-10-17T02:17:58.839 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40218.log.gz 2023-10-17T02:17:58.839 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44279.log 2023-10-17T02:17:58.839 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38542.log.gz 2023-10-17T02:17:58.839 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49633.log 2023-10-17T02:17:58.840 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52143.log.gz 2023-10-17T02:17:58.840 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.44279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49328.log 2023-10-17T02:17:58.840 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44279.log.gz 2023-10-17T02:17:58.841 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47860.log 2023-10-17T02:17:58.841 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.49633.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49633.log.gz 2023-10-17T02:17:58.841 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.49328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49418.log 2023-10-17T02:17:58.841 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49328.log.gz 2023-10-17T02:17:58.842 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.47860.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.47860.log.gz 2023-10-17T02:17:58.842 INFO:teuthology.orchestra.run.smithi169.stderr: --verbose -- /var/log/ceph/ceph-client.admin.50055.log 2023-10-17T02:17:58.842 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.49418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48194.log 2023-10-17T02:17:58.842 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49418.log.gz 2023-10-17T02:17:58.843 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.50055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45754.log 2023-10-17T02:17:58.843 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50055.log.gz 2023-10-17T02:17:58.843 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.48194.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-17T02:17:58.844 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48194.log.gz 2023-10-17T02:17:58.844 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.45754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46457.log 2023-10-17T02:17:58.844 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45754.log.gz 2023-10-17T02:17:58.844 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38974.log 2023-10-17T02:17:58.844 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.46457.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46457.log.gz 2023-10-17T02:17:58.845 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40684.log 2023-10-17T02:17:58.845 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50908.log 2023-10-17T02:17:58.845 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38974.log.gz 2023-10-17T02:17:58.846 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.40684.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40684.log.gz 2023-10-17T02:17:58.846 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44803.log 2023-10-17T02:17:58.846 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.50908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44033.log 2023-10-17T02:17:58.847 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50908.log.gz 2023-10-17T02:17:58.847 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.44803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55494.log 2023-10-17T02:17:58.848 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44803.log.gz 2023-10-17T02:17:58.848 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.44033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41292.log 2023-10-17T02:17:58.848 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44033.log.gz 2023-10-17T02:17:58.848 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38758.log 2023-10-17T02:17:58.849 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.55494.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55494.log.gz 2023-10-17T02:17:58.849 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48011.log 2023-10-17T02:17:58.849 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.41292.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41292.log.gz 2023-10-17T02:17:58.850 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44094.log 2023-10-17T02:17:58.850 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38758.log.gz 2023-10-17T02:17:58.851 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51716.log 2023-10-17T02:17:58.851 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.48011.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48011.log.gz 2023-10-17T02:17:58.852 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.44094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53753.log 2023-10-17T02:17:58.852 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44094.log.gz 2023-10-17T02:17:58.853 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48745.log 2023-10-17T02:17:58.853 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51716.log.gz 2023-10-17T02:17:58.854 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48164.log 2023-10-17T02:17:58.854 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53753.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53753.log.gz 2023-10-17T02:17:58.855 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54686.log 2023-10-17T02:17:58.855 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.48745.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48745.log.gz 2023-10-17T02:17:58.856 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.48164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40356.log 2023-10-17T02:17:58.856 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48164.log.gz 2023-10-17T02:17:58.857 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39298.log 2023-10-17T02:17:58.857 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.54686.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54686.log.gz 2023-10-17T02:17:58.858 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.40356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38778.log 2023-10-17T02:17:58.858 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40356.log.gz 2023-10-17T02:17:58.859 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39443.log 2023-10-17T02:17:58.859 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39298.log.gz 2023-10-17T02:17:58.860 INFO:teuthology.orchestra.run.smithi169.stderr:gzip/var/log/ceph/ceph-client.admin.38778.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43455.log 2023-10-17T02:17:58.860 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38778.log.gz 2023-10-17T02:17:58.860 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41617.log 2023-10-17T02:17:58.861 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39443.log.gz 2023-10-17T02:17:58.861 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.43455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40465.log 2023-10-17T02:17:58.862 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43455.log.gz 2023-10-17T02:17:58.862 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.41617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51734.log 2023-10-17T02:17:58.863 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41617.log.gz 2023-10-17T02:17:58.863 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.40465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53115.log 2023-10-17T02:17:58.864 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40465.log.gz 2023-10-17T02:17:58.864 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51734.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51734.log.gz 2023-10-17T02:17:58.865 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50485.log 2023-10-17T02:17:58.865 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46031.log 2023-10-17T02:17:58.866 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53115.log.gz 2023-10-17T02:17:58.866 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39173.log 2023-10-17T02:17:58.867 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.50485.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50485.log.gz 2023-10-17T02:17:58.867 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.46031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43572.log 2023-10-17T02:17:58.868 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46031.log.gz 2023-10-17T02:17:58.868 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52956.log 2023-10-17T02:17:58.868 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39173.log.gz 2023-10-17T02:17:58.869 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.43572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54526.log 2023-10-17T02:17:58.869 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43572.log.gz 2023-10-17T02:17:58.870 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41201.log 2023-10-17T02:17:58.870 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52956.log.gz 2023-10-17T02:17:58.871 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.54526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52690.log 2023-10-17T02:17:58.871 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54526.log.gz 2023-10-17T02:17:58.872 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.41201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49783.log 2023-10-17T02:17:58.872 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41201.log.gz 2023-10-17T02:17:58.873 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52690.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52690.log.gz 2023-10-17T02:17:58.873 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41889.log 2023-10-17T02:17:58.874 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.49783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34539.log 2023-10-17T02:17:58.874 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49783.log.gz 2023-10-17T02:17:58.875 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.41889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53859.log 2023-10-17T02:17:58.875 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41889.log.gz 2023-10-17T02:17:58.876 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.34539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54633.log 2023-10-17T02:17:58.876 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34539.log.gz 2023-10-17T02:17:58.877 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43512.log 2023-10-17T02:17:58.877 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53859.log.gz 2023-10-17T02:17:58.878 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.54633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45207.log 2023-10-17T02:17:58.878 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54633.log.gz 2023-10-17T02:17:58.878 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.43512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44939.log 2023-10-17T02:17:58.879 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43512.log.gz 2023-10-17T02:17:58.879 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.45207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45265.log 2023-10-17T02:17:58.880 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45207.log.gz 2023-10-17T02:17:58.880 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.44939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44063.log 2023-10-17T02:17:58.881 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44939.log.gz 2023-10-17T02:17:58.881 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.45265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46397.log 2023-10-17T02:17:58.881 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45265.log.gz 2023-10-17T02:17:58.882 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.44063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46193.log 2023-10-17T02:17:58.882 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44063.log.gz 2023-10-17T02:17:58.882 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.46397.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46397.log.gz 2023-10-17T02:17:58.882 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45296.log 2023-10-17T02:17:58.883 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.46193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45110.log 2023-10-17T02:17:58.883 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46193.log.gz 2023-10-17T02:17:58.883 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.45296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52480.log 2023-10-17T02:17:58.884 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45296.log.gz 2023-10-17T02:17:58.884 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.45110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42984.log 2023-10-17T02:17:58.884 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45110.log.gz 2023-10-17T02:17:58.884 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41669.log 2023-10-17T02:17:58.884 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52480.log.gz 2023-10-17T02:17:58.885 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.42984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54605.log 2023-10-17T02:17:58.885 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42984.log.gz 2023-10-17T02:17:58.885 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.41669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49723.log 2023-10-17T02:17:58.886 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41669.log.gz 2023-10-17T02:17:58.886 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.54605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40082.log 2023-10-17T02:17:58.886 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54605.log.gz 2023-10-17T02:17:58.886 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.49723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38812.log 2023-10-17T02:17:58.887 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49723.log.gz 2023-10-17T02:17:58.887 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.40082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38397.log 2023-10-17T02:17:58.887 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40082.log.gz 2023-10-17T02:17:58.887 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42460.log 2023-10-17T02:17:58.888 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38812.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38812.log.gz 2023-10-17T02:17:58.888 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43790.log 2023-10-17T02:17:58.888 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38397.log.gz 2023-10-17T02:17:58.888 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.42460.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42460.log.gz 2023-10-17T02:17:58.889 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38886.log 2023-10-17T02:17:58.889 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42189.log 2023-10-17T02:17:58.889 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.43790.log: /var/log/ceph/ceph-client.admin.38886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52878.log 2023-10-17T02:17:58.890 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38886.log.gz 2023-10-17T02:17:58.890 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43790.log.gz 2023-10-17T02:17:58.890 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.42189.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42189.log.gz 2023-10-17T02:17:58.890 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43850.log 2023-10-17T02:17:58.891 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53807.log 2023-10-17T02:17:58.891 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52878.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52878.log.gz 2023-10-17T02:17:58.891 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.43850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53568.log 2023-10-17T02:17:58.891 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43850.log.gz 2023-10-17T02:17:58.892 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53807.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-17T02:17:58.892 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53807.log.gz 2023-10-17T02:17:58.892 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47127.log 2023-10-17T02:17:58.893 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53568.log.gz 2023-10-17T02:17:58.893 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41394.log 2023-10-17T02:17:58.893 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.47127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40802.log 2023-10-17T02:17:58.893 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47127.log.gz 2023-10-17T02:17:58.894 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.41394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48134.log 2023-10-17T02:17:58.894 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41394.log.gz 2023-10-17T02:17:58.894 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43542.log 2023-10-17T02:17:58.894 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.40802.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40802.log.gz 2023-10-17T02:17:58.895 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.48134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49110.log 2023-10-17T02:17:58.895 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48134.log.gz 2023-10-17T02:17:58.895 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39082.log 2023-10-17T02:17:58.896 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.43542.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43542.log.gz 2023-10-17T02:17:58.896 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.49110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45050.log 2023-10-17T02:17:58.896 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49110.log.gz 2023-10-17T02:17:58.896 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53035.log 2023-10-17T02:17:58.897 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39082.log.gz 2023-10-17T02:17:58.897 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46671.log 2023-10-17T02:17:58.897 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.45050.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45050.log.gz 2023-10-17T02:17:58.898 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50424.log 2023-10-17T02:17:58.898 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53035.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53035.log.gz 2023-10-17T02:17:58.898 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.46671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50393.log 2023-10-17T02:17:58.898 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46671.log.gz 2023-10-17T02:17:58.899 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.50424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42771.log 2023-10-17T02:17:58.899 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50424.log.gz 2023-10-17T02:17:58.899 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.50393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51996.log 2023-10-17T02:17:58.900 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50393.log.gz 2023-10-17T02:17:58.900 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.42771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42093.log 2023-10-17T02:17:58.900 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42771.log.gz 2023-10-17T02:17:58.900 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51996.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-17T02:17:58.901 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51996.log.gz 2023-10-17T02:17:58.901 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.42093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51868.log 2023-10-17T02:17:58.901 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42093.log.gz 2023-10-17T02:17:58.902 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44626.log 2023-10-17T02:17:58.902 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.51868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40495.log 2023-10-17T02:17:58.902 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51868.log.gz 2023-10-17T02:17:58.903 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.44626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43482.log 2023-10-17T02:17:58.903 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44626.log.gz 2023-10-17T02:17:58.903 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.40495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39136.log 2023-10-17T02:17:58.903 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40495.log.gz 2023-10-17T02:17:58.904 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.43482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42801.log 2023-10-17T02:17:58.904 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43482.log.gz 2023-10-17T02:17:58.904 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51792.log 2023-10-17T02:17:58.905 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39136.log.gz 2023-10-17T02:17:58.905 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48254.log 2023-10-17T02:17:58.905 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.42801.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42801.log.gz 2023-10-17T02:17:58.905 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39578.log 2023-10-17T02:17:58.906 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51792.log.gz 2023-10-17T02:17:58.906 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.48254.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48254.log.gz 2023-10-17T02:17:58.906 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49019.log 2023-10-17T02:17:58.907 INFO:teuthology.orchestra.run.smithi169.stderr:gzip/var/log/ceph/ceph-client.admin.39578.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.52532.log 2023-10-17T02:17:58.907 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39578.log.gz 2023-10-17T02:17:58.907 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.49019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54097.log 2023-10-17T02:17:58.908 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49019.log.gz 2023-10-17T02:17:58.908 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41726.log 2023-10-17T02:17:58.908 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52532.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52532.log.gz 2023-10-17T02:17:58.909 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.54097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48224.log 2023-10-17T02:17:58.909 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54097.log.gz 2023-10-17T02:17:58.909 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.41726.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41726.log.gz 2023-10-17T02:17:58.909 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40772.log 2023-10-17T02:17:58.910 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.48224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54393.log 2023-10-17T02:17:58.910 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48224.log.gz 2023-10-17T02:17:58.910 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.40772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39406.log 2023-10-17T02:17:58.911 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40772.log.gz 2023-10-17T02:17:58.911 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.54393.log: /var/log/ceph/ceph-client.admin.53356.log 2023-10-17T02:17:58.911 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54393.log.gz 2023-10-17T02:17:58.911 INFO:teuthology.orchestra.run.smithi169.stderr:gzip/var/log/ceph/ceph-client.admin.39406.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.46611.log 2023-10-17T02:17:58.912 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39406.log.gz 2023-10-17T02:17:58.912 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39011.log 2023-10-17T02:17:58.912 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53356.log.gz 2023-10-17T02:17:58.913 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37887.log 2023-10-17T02:17:58.913 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.46611.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46611.log.gz 2023-10-17T02:17:58.913 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39011.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39011.log.gz 2023-10-17T02:17:58.913 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51940.log 2023-10-17T02:17:58.914 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49753.log 2023-10-17T02:17:58.914 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.37887.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37887.log.gz 2023-10-17T02:17:58.914 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51752.log 2023-10-17T02:17:58.915 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51940.log.gz 2023-10-17T02:17:58.915 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.49753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51547.log 2023-10-17T02:17:58.915 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49753.log.gz 2023-10-17T02:17:58.915 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44656.log 2023-10-17T02:17:58.916 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51752.log.gz 2023-10-17T02:17:58.916 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51547.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.51547.log.gz 2023-10-17T02:17:58.916 INFO:teuthology.orchestra.run.smithi169.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47343.log 2023-10-17T02:17:58.917 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.44656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51092.log 2023-10-17T02:17:58.917 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44656.log.gz 2023-10-17T02:17:58.917 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.47343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48437.log 2023-10-17T02:17:58.917 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47343.log.gz 2023-10-17T02:17:58.918 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51092.log: 90.0% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-17T02:17:58.918 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51092.log.gz 2023-10-17T02:17:58.918 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39914.log 2023-10-17T02:17:58.919 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42831.log 2023-10-17T02:17:58.919 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.48437.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48437.log.gz 2023-10-17T02:17:58.919 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39318.log 2023-10-17T02:17:58.919 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39914.log.gz 2023-10-17T02:17:58.920 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.42831.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42831.log.gzgzip 2023-10-17T02:17:58.920 INFO:teuthology.orchestra.run.smithi169.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47617.log 2023-10-17T02:17:58.920 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40408.log 2023-10-17T02:17:58.920 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39318.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39318.log.gz 2023-10-17T02:17:58.921 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.47617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48776.log 2023-10-17T02:17:58.921 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47617.log.gz 2023-10-17T02:17:58.921 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.40408.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.47223.log 2023-10-17T02:17:58.921 INFO:teuthology.orchestra.run.smithi169.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40408.log.gz 2023-10-17T02:17:58.922 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49541.log 2023-10-17T02:17:58.922 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.48776.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48776.log.gz 2023-10-17T02:17:58.922 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.47223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48467.log 2023-10-17T02:17:58.922 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47223.log.gz 2023-10-17T02:17:58.923 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.49541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54069.log 2023-10-17T02:17:58.923 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49541.log.gz 2023-10-17T02:17:58.923 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.48467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51306.log 2023-10-17T02:17:58.923 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48467.log.gz 2023-10-17T02:17:58.924 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.54069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55196.log 2023-10-17T02:17:58.924 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54069.log.gz 2023-10-17T02:17:58.924 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45844.log 2023-10-17T02:17:58.925 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51306.log.gz 2023-10-17T02:17:58.925 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.55196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48072.log 2023-10-17T02:17:58.925 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55196.log.gz 2023-10-17T02:17:58.925 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.45844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48346.log 2023-10-17T02:17:58.926 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45844.log.gz 2023-10-17T02:17:58.926 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.48072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42336.log 2023-10-17T02:17:58.926 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48072.log.gz 2023-10-17T02:17:58.927 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.48346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41543.log 2023-10-17T02:17:58.927 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48346.log.gz 2023-10-17T02:17:58.927 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.42336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53542.log 2023-10-17T02:17:58.928 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42336.log.gz 2023-10-17T02:17:58.928 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.41543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42553.log 2023-10-17T02:17:58.928 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41543.log.gz 2023-10-17T02:17:58.928 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42216.log 2023-10-17T02:17:58.929 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53542.log.gz 2023-10-17T02:17:58.929 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.42553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43431.log 2023-10-17T02:17:58.929 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42553.log.gz 2023-10-17T02:17:58.930 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.42216.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.42216.log.gz --verbose 2023-10-17T02:17:58.930 INFO:teuthology.orchestra.run.smithi169.stderr: -- /var/log/ceph/ceph-client.admin.52664.log 2023-10-17T02:17:58.930 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.43431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49843.log 2023-10-17T02:17:58.930 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43431.log.gz 2023-10-17T02:17:58.931 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52664.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52664.log.gz 2023-10-17T02:17:58.931 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43946.log 2023-10-17T02:17:58.931 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40931.log 2023-10-17T02:17:58.931 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.49843.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49843.log.gz 2023-10-17T02:17:58.932 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.43946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45080.log 2023-10-17T02:17:58.932 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43946.log.gz 2023-10-17T02:17:58.932 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.40931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52982.log 2023-10-17T02:17:58.932 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40931.log.gz 2023-10-17T02:17:58.933 INFO:teuthology.orchestra.run.smithi169.stderr:gzip/var/log/ceph/ceph-client.admin.45080.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.47587.log 2023-10-17T02:17:58.933 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45080.log.gz 2023-10-17T02:17:58.933 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55253.log 2023-10-17T02:17:58.933 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52982.log.gz 2023-10-17T02:17:58.934 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.47587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50878.log 2023-10-17T02:17:58.934 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47587.log.gz 2023-10-17T02:17:58.934 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.55253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46886.log 2023-10-17T02:17:58.935 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55253.log.gz 2023-10-17T02:17:58.935 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45182.log 2023-10-17T02:17:58.935 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.50878.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50878.log.gz 2023-10-17T02:17:58.935 INFO:teuthology.orchestra.run.smithi169.stderr:gzip/var/log/ceph/ceph-client.admin.46886.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.50665.log 2023-10-17T02:17:58.936 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46886.log.gz 2023-10-17T02:17:58.936 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.45182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44003.log 2023-10-17T02:17:58.936 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45182.log.gz 2023-10-17T02:17:58.936 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55116.log 2023-10-17T02:17:58.937 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.50665.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50665.log.gz 2023-10-17T02:17:58.937 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.44003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41913.log 2023-10-17T02:17:58.937 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44003.log.gz 2023-10-17T02:17:58.938 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.55116.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55116.log.gz -5 2023-10-17T02:17:58.938 INFO:teuthology.orchestra.run.smithi169.stderr: --verbose -- /var/log/ceph/ceph-client.admin.43046.log 2023-10-17T02:17:58.938 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.41913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51608.log 2023-10-17T02:17:58.938 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41913.log.gz 2023-10-17T02:17:58.939 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.43046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39610.log 2023-10-17T02:17:58.939 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43046.log.gz 2023-10-17T02:17:58.939 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54959.log 2023-10-17T02:17:58.939 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51608.log.gz 2023-10-17T02:17:58.939 INFO:teuthology.orchestra.run.smithi169.stderr:gzip/var/log/ceph/ceph-client.admin.39610.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34515.log 2023-10-17T02:17:58.940 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39610.log.gz 2023-10-17T02:17:58.940 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.54959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50575.log 2023-10-17T02:17:58.940 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54959.log.gz 2023-10-17T02:17:58.940 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.34515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50605.log 2023-10-17T02:17:58.941 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34515.log.gz 2023-10-17T02:17:58.941 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.50575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46336.log 2023-10-17T02:17:58.941 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50575.log.gz 2023-10-17T02:17:58.942 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.50605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46275.log 2023-10-17T02:17:58.942 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50605.log.gz 2023-10-17T02:17:58.942 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41857.log 2023-10-17T02:17:58.942 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.46336.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46336.log.gz 2023-10-17T02:17:58.943 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.46275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43602.log 2023-10-17T02:17:58.943 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46275.log.gz 2023-10-17T02:17:58.943 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.41857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39498.log 2023-10-17T02:17:58.944 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41857.log.gz 2023-10-17T02:17:58.944 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.43602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40040.log 2023-10-17T02:17:58.945 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43602.log.gz 2023-10-17T02:17:58.945 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39498.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.39498.log.gz 2023-10-17T02:17:58.945 INFO:teuthology.orchestra.run.smithi169.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45971.log 2023-10-17T02:17:58.945 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54123.log 2023-10-17T02:17:58.946 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.40040.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40040.log.gz 2023-10-17T02:17:58.946 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.45971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55414.log 2023-10-17T02:17:58.946 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45971.log.gz 2023-10-17T02:17:58.947 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.54123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50332.log 2023-10-17T02:17:58.947 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54123.log.gz 2023-10-17T02:17:58.947 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46217.log 2023-10-17T02:17:58.947 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.55414.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55414.log.gz 2023-10-17T02:17:58.948 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.50332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50971.log 2023-10-17T02:17:58.948 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50332.log.gz 2023-10-17T02:17:58.948 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51366.log 2023-10-17T02:17:58.949 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.46217.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46217.log.gz 2023-10-17T02:17:58.949 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37810.log 2023-10-17T02:17:58.949 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.50971.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50971.log.gz 2023-10-17T02:17:58.950 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52904.log 2023-10-17T02:17:58.950 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51366.log.gz 2023-10-17T02:17:58.950 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.37810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42957.log 2023-10-17T02:17:58.951 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37810.log.gz 2023-10-17T02:17:58.951 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49813.log 2023-10-17T02:17:58.951 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52904.log.gz 2023-10-17T02:17:58.951 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.42957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40879.log 2023-10-17T02:17:58.952 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42957.log.gz 2023-10-17T02:17:58.952 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.49813.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.49813.log.gz -- 2023-10-17T02:17:58.952 INFO:teuthology.orchestra.run.smithi169.stderr: /var/log/ceph/ceph-client.admin.47981.log 2023-10-17T02:17:58.952 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39227.log 2023-10-17T02:17:58.953 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.40879.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40879.log.gz 2023-10-17T02:17:58.953 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.47981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45645.log 2023-10-17T02:17:58.953 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47981.log.gz 2023-10-17T02:17:58.953 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48531.log 2023-10-17T02:17:58.954 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39227.log.gz 2023-10-17T02:17:58.954 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.45645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53649.log 2023-10-17T02:17:58.954 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45645.log.gz 2023-10-17T02:17:58.955 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50455.log 2023-10-17T02:17:58.955 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.48531.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48531.log.gz 2023-10-17T02:17:58.955 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48956.log 2023-10-17T02:17:58.955 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53649.log.gz 2023-10-17T02:17:58.956 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54743.log 2023-10-17T02:17:58.956 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.50455.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50455.log.gz 2023-10-17T02:17:58.956 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.48956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41122.log 2023-10-17T02:17:58.957 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48956.log.gz 2023-10-17T02:17:58.957 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.54743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43973.log 2023-10-17T02:17:58.957 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54743.log.gz 2023-10-17T02:17:58.957 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.41122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41787.log 2023-10-17T02:17:58.958 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41122.log.gz 2023-10-17T02:17:58.958 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.43973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53410.log 2023-10-17T02:17:58.958 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43973.log.gz 2023-10-17T02:17:58.959 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.41787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49297.log 2023-10-17T02:17:58.959 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41787.log.gz 2023-10-17T02:17:58.959 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37412.log 2023-10-17T02:17:58.959 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53410.log.gz 2023-10-17T02:17:58.960 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.49297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45356.log 2023-10-17T02:17:58.960 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49297.log.gz 2023-10-17T02:17:58.960 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.37412.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-17T02:17:58.960 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37412.log.gz 2023-10-17T02:17:58.960 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.45356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52452.log 2023-10-17T02:17:58.961 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45356.log.gz 2023-10-17T02:17:58.961 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48407.log 2023-10-17T02:17:58.961 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39998.log 2023-10-17T02:17:58.962 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52452.log.gz 2023-10-17T02:17:58.962 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.48407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39830.log 2023-10-17T02:17:58.962 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48407.log.gz 2023-10-17T02:17:58.962 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45890.log 2023-10-17T02:17:58.963 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39998.log.gz 2023-10-17T02:17:58.963 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40435.log 2023-10-17T02:17:58.963 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39830.log.gz 2023-10-17T02:17:58.964 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.45890.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.45890.log.gz -5 2023-10-17T02:17:58.964 INFO:teuthology.orchestra.run.smithi169.stderr: --verbose -- /var/log/ceph/ceph-client.admin.45571.log 2023-10-17T02:17:58.964 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.40435.log: -- /var/log/ceph/ceph-client.admin.38940.log 2023-10-17T02:17:58.964 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40435.log.gz 2023-10-17T02:17:58.965 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.45571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54876.log 2023-10-17T02:17:58.965 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45571.log.gz 2023-10-17T02:17:58.965 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51771.log 2023-10-17T02:17:58.965 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38940.log.gz 2023-10-17T02:17:58.966 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.54876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42003.log 2023-10-17T02:17:58.966 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54876.log.gz 2023-10-17T02:17:58.966 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48041.log 2023-10-17T02:17:58.967 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51771.log.gz 2023-10-17T02:17:58.967 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.42003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50147.log 2023-10-17T02:17:58.967 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42003.log.gz 2023-10-17T02:17:58.967 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49935.log 2023-10-17T02:17:58.968 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.48041.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48041.log.gz 2023-10-17T02:17:58.968 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.50147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49261.log 2023-10-17T02:17:58.968 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50147.log.gz 2023-10-17T02:17:58.968 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.49935.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49935.log.gz 2023-10-17T02:17:58.969 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40166.log 2023-10-17T02:17:58.969 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53193.log 2023-10-17T02:17:58.969 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.49261.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49261.log.gz 2023-10-17T02:17:58.970 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.40166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42033.log 2023-10-17T02:17:58.970 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40166.log.gz 2023-10-17T02:17:58.970 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53991.log 2023-10-17T02:17:58.970 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53193.log.gz 2023-10-17T02:17:58.971 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.42033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53329.log 2023-10-17T02:17:58.971 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42033.log.gz 2023-10-17T02:17:58.971 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49201.log 2023-10-17T02:17:58.972 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53991.log.gz 2023-10-17T02:17:58.972 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47066.log 2023-10-17T02:17:58.972 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53329.log.gz 2023-10-17T02:17:58.973 INFO:teuthology.orchestra.run.smithi169.stderr:gzip/var/log/ceph/ceph-client.admin.49201.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.46946.log 2023-10-17T02:17:58.973 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49201.log.gz 2023-10-17T02:17:58.973 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.47066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41696.log 2023-10-17T02:17:58.973 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47066.log.gz 2023-10-17T02:17:58.974 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.46946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39480.log 2023-10-17T02:17:58.974 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46946.log.gz 2023-10-17T02:17:58.974 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.41696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37200.log 2023-10-17T02:17:58.975 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41696.log.gz 2023-10-17T02:17:58.975 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46976.log 2023-10-17T02:17:58.975 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39480.log.gz 2023-10-17T02:17:58.975 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.37200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48836.log 2023-10-17T02:17:58.976 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37200.log.gz 2023-10-17T02:17:58.976 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.46976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44554.log 2023-10-17T02:17:58.976 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46976.log.gz 2023-10-17T02:17:58.976 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.48836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43238.log 2023-10-17T02:17:58.977 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48836.log.gz 2023-10-17T02:17:58.977 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.44554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39264.log 2023-10-17T02:17:58.977 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44554.log.gz 2023-10-17T02:17:58.977 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39210.log 2023-10-17T02:17:58.978 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.43238.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43238.log.gz 2023-10-17T02:17:58.978 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46791.log 2023-10-17T02:17:58.978 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39264.log.gz 2023-10-17T02:17:58.979 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46091.log 2023-10-17T02:17:58.979 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39210.log.gz 2023-10-17T02:17:58.979 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.46791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47191.log 2023-10-17T02:17:58.980 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46791.log.gz 2023-10-17T02:17:58.980 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.46091.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.30838.log 2023-10-17T02:17:58.980 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46091.log.gz 2023-10-17T02:17:58.980 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.47191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46580.log 2023-10-17T02:17:58.981 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47191.log.gz 2023-10-17T02:17:58.981 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph.tmp-client.admin.30838.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.30838.log.gz 2023-10-17T02:17:58.981 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51152.log 2023-10-17T02:17:58.982 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.46580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42657.log 2023-10-17T02:17:58.982 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46580.log.gz 2023-10-17T02:17:58.982 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46306.log 2023-10-17T02:17:58.983 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51152.log.gz 2023-10-17T02:17:58.983 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.42657.log: /var/log/ceph/ceph-client.admin.40124.log 2023-10-17T02:17:58.983 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42657.log.gz 2023-10-17T02:17:58.983 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.46306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44411.log 2023-10-17T02:17:58.984 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46306.log.gz 2023-10-17T02:17:58.984 INFO:teuthology.orchestra.run.smithi169.stderr:gzip/var/log/ceph/ceph-client.admin.40124.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38562.log 2023-10-17T02:17:58.984 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40124.log.gz 2023-10-17T02:17:58.984 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52586.log 2023-10-17T02:17:58.985 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.44411.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44411.log.gz 2023-10-17T02:17:58.985 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50515.log 2023-10-17T02:17:58.985 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38562.log.gz 2023-10-17T02:17:58.986 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39703.log 2023-10-17T02:17:58.986 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52586.log.gz 2023-10-17T02:17:58.986 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.50515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52426.log 2023-10-17T02:17:58.987 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50515.log.gz 2023-10-17T02:17:58.987 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53382.log 2023-10-17T02:17:58.987 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39703.log.gz 2023-10-17T02:17:58.988 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39335.log 2023-10-17T02:17:58.988 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52426.log.gz 2023-10-17T02:17:58.988 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41450.log 2023-10-17T02:17:58.988 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53382.log.gz 2023-10-17T02:17:58.989 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46244.log 2023-10-17T02:17:58.989 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39335.log.gz 2023-10-17T02:17:58.989 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.41450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49480.log 2023-10-17T02:17:58.989 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41450.log.gz 2023-10-17T02:17:58.990 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.46244.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46244.log.gz 2023-10-17T02:17:58.990 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44221.log 2023-10-17T02:17:58.990 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.49480.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49480.log.gz 2023-10-17T02:17:58.991 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44494.log 2023-10-17T02:17:58.991 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.44221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46061.log 2023-10-17T02:17:58.991 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44221.log.gz 2023-10-17T02:17:58.992 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.44494.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.44494.log.gz 2023-10-17T02:17:58.992 INFO:teuthology.orchestra.run.smithi169.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.46518.log 2023-10-17T02:17:58.992 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46916.log 2023-10-17T02:17:58.992 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.46061.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46061.log.gz 2023-10-17T02:17:58.993 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.46518.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46518.log.gz 2023-10-17T02:17:58.993 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43359.log 2023-10-17T02:17:58.993 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37635.log 2023-10-17T02:17:58.994 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.46916.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46916.log.gz 2023-10-17T02:17:58.994 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.43359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43922.log 2023-10-17T02:17:58.994 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43359.log.gz 2023-10-17T02:17:58.995 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.37635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47374.log 2023-10-17T02:17:58.995 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37635.log.gz 2023-10-17T02:17:58.995 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.43922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44169.log 2023-10-17T02:17:58.995 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43922.log.gz 2023-10-17T02:17:58.996 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.47374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45154.log 2023-10-17T02:17:58.996 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47374.log.gz 2023-10-17T02:17:58.996 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.44169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40660.log 2023-10-17T02:17:58.997 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44169.log.gz 2023-10-17T02:17:58.997 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.45154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54150.log 2023-10-17T02:17:58.997 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45154.log.gz 2023-10-17T02:17:58.998 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.40660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49995.log 2023-10-17T02:17:58.998 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40660.log.gz 2023-10-17T02:17:58.998 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.54150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47556.log 2023-10-17T02:17:58.998 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54150.log.gz 2023-10-17T02:17:58.999 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.49995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52034.log 2023-10-17T02:17:58.999 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49995.log.gz 2023-10-17T02:17:58.999 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.47556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53965.log 2023-10-17T02:17:58.999 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47556.log.gz 2023-10-17T02:17:59.000 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43329.log 2023-10-17T02:17:59.000 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52034.log.gz 2023-10-17T02:17:59.000 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54366.log 2023-10-17T02:17:59.001 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53965.log.gz 2023-10-17T02:17:59.001 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.43329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54230.log 2023-10-17T02:17:59.001 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43329.log.gz 2023-10-17T02:17:59.001 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.54366.log: /var/log/ceph/ceph-client.admin.41756.log 2023-10-17T02:17:59.002 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54366.log.gz 2023-10-17T02:17:59.002 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.54230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51698.log 2023-10-17T02:17:59.002 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54230.log.gz 2023-10-17T02:17:59.003 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47006.log 2023-10-17T02:17:59.003 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.41756.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41756.log.gz 2023-10-17T02:17:59.003 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38994.log 2023-10-17T02:17:59.003 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51698.log.gz 2023-10-17T02:17:59.004 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53675.log 2023-10-17T02:17:59.004 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.47006.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47006.log.gz 2023-10-17T02:17:59.004 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47920.log 2023-10-17T02:17:59.004 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38994.log.gz 2023-10-17T02:17:59.005 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44963.log 2023-10-17T02:17:59.005 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53675.log.gz 2023-10-17T02:17:59.005 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.47920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47036.log 2023-10-17T02:17:59.006 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47920.log.gz 2023-10-17T02:17:59.006 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.44963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55444.log 2023-10-17T02:17:59.006 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44963.log.gz 2023-10-17T02:17:59.007 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.47036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54419.log 2023-10-17T02:17:59.007 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47036.log.gz 2023-10-17T02:17:59.007 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.55444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45724.log 0.0% 2023-10-17T02:17:59.008 INFO:teuthology.orchestra.run.smithi169.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55444.log.gz 2023-10-17T02:17:59.008 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.54419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48896.log 2023-10-17T02:17:59.008 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54419.log.gz 2023-10-17T02:17:59.008 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.45724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44910.log 2023-10-17T02:17:59.009 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45724.log.gz 2023-10-17T02:17:59.009 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.48896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45541.log 2023-10-17T02:17:59.009 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48896.log.gz 2023-10-17T02:17:59.009 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.44910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42871.log 2023-10-17T02:17:59.010 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44910.log.gz 2023-10-17T02:17:59.010 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.45541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47313.log 2023-10-17T02:17:59.010 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45541.log.gz 2023-10-17T02:17:59.010 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.42871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50301.log 2023-10-17T02:17:59.011 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42871.log.gz 2023-10-17T02:17:59.011 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.47313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44682.log 2023-10-17T02:17:59.011 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47313.log.gz 2023-10-17T02:17:59.012 INFO:teuthology.orchestra.run.smithi169.stderr:gzip/var/log/ceph/ceph-client.admin.50301.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.51577.log 2023-10-17T02:17:59.012 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50301.log.gz 2023-10-17T02:17:59.012 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.44682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38832.log 2023-10-17T02:17:59.012 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44682.log.gz 2023-10-17T02:17:59.013 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51577.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51577.log.gz 2023-10-17T02:17:59.013 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39028.log 2023-10-17T02:17:59.013 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38704.log 2023-10-17T02:17:59.013 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38832.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38832.log.gz 2023-10-17T02:17:59.014 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55038.log 2023-10-17T02:17:59.014 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39028.log.gz 2023-10-17T02:17:59.014 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53219.log 2023-10-17T02:17:59.015 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38704.log.gz 2023-10-17T02:17:59.015 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50756.log 2023-10-17T02:17:59.015 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.55038.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55038.log.gz 2023-10-17T02:17:59.016 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53219.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53219.log.gz 2023-10-17T02:17:59.016 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41366.log 2023-10-17T02:17:59.016 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46001.log 2023-10-17T02:17:59.016 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.50756.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50756.log.gz 2023-10-17T02:17:59.017 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.41366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37628.log 2023-10-17T02:17:59.017 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41366.log.gz 2023-10-17T02:17:59.018 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.46001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49693.log 2023-10-17T02:17:59.018 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46001.log.gz 2023-10-17T02:17:59.019 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.37628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43759.log 2023-10-17T02:17:59.019 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37628.log.gz 2023-10-17T02:17:59.020 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47157.log 2023-10-17T02:17:59.020 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.49693.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49693.log.gz 2023-10-17T02:17:59.021 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.43759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44584.log 2023-10-17T02:17:59.021 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43759.log.gz 2023-10-17T02:17:59.022 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.47157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53727.log 2023-10-17T02:17:59.022 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47157.log.gz 2023-10-17T02:17:59.022 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.44584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41817.log 2023-10-17T02:17:59.023 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44584.log.gz 2023-10-17T02:17:59.023 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52322.log 2023-10-17T02:17:59.023 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53727.log.gz 2023-10-17T02:17:59.023 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.41817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46367.log 2023-10-17T02:17:59.024 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41817.log.gz 2023-10-17T02:17:59.024 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37558.log 2023-10-17T02:17:59.024 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52322.log.gz 2023-10-17T02:17:59.024 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.46367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52166.log 2023-10-17T02:17:59.025 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46367.log.gz 2023-10-17T02:17:59.025 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.37558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46165.log 2023-10-17T02:17:59.025 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37558.log.gz 2023-10-17T02:17:59.026 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52166.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52166.log.gz 2023-10-17T02:17:59.026 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52070.log 2023-10-17T02:17:59.026 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38488.log 2023-10-17T02:17:59.027 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.46165.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46165.log.gz 2023-10-17T02:17:59.027 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45918.log 2023-10-17T02:17:59.027 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52070.log: /var/log/ceph/ceph-client.admin.38488.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52070.log.gz 2023-10-17T02:17:59.027 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38488.log.gz 2023-10-17T02:17:59.028 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51457.log 2023-10-17T02:17:59.028 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.45918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39119.log 2023-10-17T02:17:59.028 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45918.log.gz 2023-10-17T02:17:59.028 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55169.log 2023-10-17T02:17:59.029 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51457.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51457.log.gz 2023-10-17T02:17:59.029 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39546.log 2023-10-17T02:17:59.029 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39119.log.gz 2023-10-17T02:17:59.029 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.55169.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55169.log.gz 2023-10-17T02:17:59.030 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40525.log 2023-10-17T02:17:59.030 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47526.log 2023-10-17T02:17:59.030 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39546.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39546.log.gz 2023-10-17T02:17:59.031 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.40525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45784.log 2023-10-17T02:17:59.031 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40525.log.gz 2023-10-17T02:17:59.031 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.47526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52107.log 2023-10-17T02:17:59.031 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47526.log.gz 2023-10-17T02:17:59.032 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48987.log 2023-10-17T02:17:59.032 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.45784.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45784.log.gz 2023-10-17T02:17:59.032 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53779.log 2023-10-17T02:17:59.033 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52107.log.gz 2023-10-17T02:17:59.033 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.48987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48593.log 2023-10-17T02:17:59.033 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48987.log.gz 2023-10-17T02:17:59.033 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5/var/log/ceph/ceph-client.admin.53779.log: --verbose -- /var/log/ceph/ceph-client.admin.42681.log 2023-10-17T02:17:59.034 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53779.log.gz 2023-10-17T02:17:59.034 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.48593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38670.log 2023-10-17T02:17:59.034 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48593.log.gz 2023-10-17T02:17:59.034 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.42681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53061.log 2023-10-17T02:17:59.035 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42681.log.gz 2023-10-17T02:17:59.035 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37784.log 2023-10-17T02:17:59.035 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38670.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38670.log.gz 2023-10-17T02:17:59.035 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44437.log 2023-10-17T02:17:59.036 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53061.log.gz 2023-10-17T02:17:59.036 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.37784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51922.log 2023-10-17T02:17:59.037 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37784.log.gz 2023-10-17T02:17:59.037 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.44437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52743.log 2023-10-17T02:17:59.037 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44437.log.gz 2023-10-17T02:17:59.037 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53623.log 2023-10-17T02:17:59.038 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51922.log.gz 2023-10-17T02:17:59.038 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.52743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46124.log 2023-10-17T02:17:59.039 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52743.log.gz 2023-10-17T02:17:59.039 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43187.log 2023-10-17T02:17:59.040 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53623.log.gz 2023-10-17T02:17:59.040 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.46124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53886.log 2023-10-17T02:17:59.040 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46124.log.gz 2023-10-17T02:17:59.041 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.43187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38596.log 2023-10-17T02:17:59.041 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43187.log.gz 2023-10-17T02:17:59.042 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51122.log 2023-10-17T02:17:59.042 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53886.log.gz 2023-10-17T02:17:59.043 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44524.log 2023-10-17T02:17:59.043 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38596.log.gz 2023-10-17T02:17:59.043 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44464.log 2023-10-17T02:17:59.043 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51122.log.gz 2023-10-17T02:17:59.044 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.44524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50635.log 2023-10-17T02:17:59.044 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44524.log.gz 2023-10-17T02:17:59.044 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.44464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47890.log 2023-10-17T02:17:59.044 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44464.log.gz 2023-10-17T02:17:59.045 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.50635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54257.log 2023-10-17T02:17:59.045 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50635.log.gz 2023-10-17T02:17:59.045 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.47890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51427.log 2023-10-17T02:17:59.045 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47890.log.gz 2023-10-17T02:17:59.046 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53701.log 2023-10-17T02:17:59.046 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.54257.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54257.log.gz 2023-10-17T02:17:59.046 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40632.log 2023-10-17T02:17:59.047 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51427.log.gz 2023-10-17T02:17:59.047 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.53701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51810.log 2023-10-17T02:17:59.047 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53701.log.gz 2023-10-17T02:17:59.047 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37557.log 2023-10-17T02:17:59.048 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.40632.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40632.log.gz 2023-10-17T02:17:59.048 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.51810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54903.log 2023-10-17T02:17:59.048 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51810.log.gz 2023-10-17T02:17:59.049 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.37557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39389.log 2023-10-17T02:17:59.049 INFO:teuthology.orchestra.run.smithi169.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37557.log.gz 2023-10-17T02:17:59.049 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.54903.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54903.log.gz 2023-10-17T02:17:59.049 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.39389.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39389.log.gz 2023-10-17T02:17:59.050 INFO:teuthology.orchestra.run.smithi169.stderr: 28.9% -- replaced with /var/log/ceph/ceph-client.admin.44710.log.gz 2023-10-17T02:17:59.051 INFO:teuthology.orchestra.run.smithi169.stderr: 90.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-17T02:18:16.138 INFO:teuthology.orchestra.run.smithi169.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-17T02:18:25.593 INFO:teuthology.orchestra.run.smithi169.stderr: 89.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-17T02:55:14.308 INFO:teuthology.orchestra.run.smithi169.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-17T03:04:45.750 INFO:teuthology.orchestra.run.smithi169.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-17T03:04:46.893 INFO:teuthology.orchestra.run.smithi169.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-17T03:04:46.895 INFO:teuthology.orchestra.run.smithi169.stderr: 2023-10-17T03:04:46.895 INFO:teuthology.orchestra.run.smithi169.stderr:real 46m48.448s 2023-10-17T03:04:46.896 INFO:teuthology.orchestra.run.smithi169.stderr:user 45m3.808s 2023-10-17T03:04:46.896 INFO:teuthology.orchestra.run.smithi169.stderr:sys 2m22.322s 2023-10-17T03:04:46.897 INFO:tasks.ceph:Archiving logs... 2023-10-17T03:04:47.024 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/log/ceph to /home/teuthworker/archive/rraja-2023-10-17_00:10:32-rbd:cli-debug-62994-distro-default-smithi/7430972/remote/smithi169/log 2023-10-17T03:04:47.026 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-17T03:06:50.665 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-17T03:06:50.680 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-17T03:06:50.681 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-17T03:06:50.732 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-17T03:06:50.746 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-17T03:06:50.747 DEBUG:teuthology.orchestra.run.smithi169:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-17T03:06:50.830 INFO:teuthology.orchestra.run.smithi169.stdout: remote refid st t when poll reach delay offset jitter 2023-10-17T03:06:50.830 INFO:teuthology.orchestra.run.smithi169.stdout:============================================================================== 2023-10-17T03:06:50.831 INFO:teuthology.orchestra.run.smithi169.stdout:*hv01.front.sepi 44.4.53.2 2 u 22 256 377 0.172 -0.502 0.259 2023-10-17T03:06:50.831 INFO:teuthology.orchestra.run.smithi169.stdout:+hv02.front.sepi 63.231.80.2 3 u 93 256 377 0.201 +1.335 0.111 2023-10-17T03:06:50.831 INFO:teuthology.orchestra.run.smithi169.stdout:+hv03.front.sepi 74.6.168.72 3 u 84 256 377 0.187 -4.323 0.053 2023-10-17T03:06:50.832 INFO:teuthology.orchestra.run.smithi169.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-10-17T03:06:50.833 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-17T03:06:50.842 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-17T03:06:50.843 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-17T03:06:50.853 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-17T03:06:50.862 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-17T03:06:50.871 INFO:teuthology.task.internal:Duration was 9277.166816 seconds 2023-10-17T03:06:50.872 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-17T03:06:50.881 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-17T03:06:50.882 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-17T03:06:51.170 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-17T03:06:51.171 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi169.front.sepia.ceph.com 2023-10-17T03:06:51.171 DEBUG:teuthology.orchestra.run.smithi169:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-17T03:06:51.247 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-17T03:06:51.248 DEBUG:teuthology.orchestra.run.smithi169:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-17T03:06:51.380 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-17T03:06:51.381 DEBUG:teuthology.orchestra.run.smithi169:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-17T03:06:51.917 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-17T03:06:51.928 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-17T03:06:51.928 DEBUG:teuthology.orchestra.run.smithi169:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-17T03:06:51.988 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-17T03:06:51.998 DEBUG:teuthology.orchestra.run.smithi169:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-17T03:06:52.029 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern = core 2023-10-17T03:06:52.066 DEBUG:teuthology.orchestra.run.smithi169:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-17T03:06:52.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-17T03:06:52.112 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-17T03:06:52.122 INFO:teuthology.task.internal:Transferring archived files... 2023-10-17T03:06:52.124 DEBUG:teuthology.misc:Transferring archived files from smithi169:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rraja-2023-10-17_00:10:32-rbd:cli-debug-62994-distro-default-smithi/7430972/remote/smithi169 2023-10-17T03:06:52.124 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-17T03:06:52.176 INFO:teuthology.task.internal:Removing archive directory... 2023-10-17T03:06:52.177 DEBUG:teuthology.orchestra.run.smithi169:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-17T03:06:52.220 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-17T03:06:52.233 INFO:teuthology.task.internal:Not uploading archives. 2023-10-17T03:06:52.233 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-17T03:06:52.244 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-17T03:06:52.244 DEBUG:teuthology.orchestra.run.smithi169:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-17T03:06:52.294 INFO:teuthology.orchestra.run.smithi169.stdout: 655472 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 17 03:06 /home/ubuntu/cephtest 2023-10-17T03:06:52.295 INFO:teuthology.orchestra.run.smithi169.stdout: 673499 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Oct 17 00:42 /home/ubuntu/cephtest/mnt.0 2023-10-17T03:06:52.295 INFO:teuthology.orchestra.run.smithi169.stdout: 673925 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Oct 17 00:43 /home/ubuntu/cephtest/mnt.0/client.0 2023-10-17T03:06:52.296 INFO:teuthology.orchestra.run.smithi169.stdout: 687832 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Oct 17 00:43 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-17T03:06:52.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-17T03:06:52.298 INFO:teuthology.orchestra.run.smithi169.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-10-17T03:06:52.298 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi169 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-10-17T03:06:52.307 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-17T03:06:52.318 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-17T03:06:52.329 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi169 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-10-17T03:06:52.330 INFO:teuthology.nuke:Checking targets against current locks 2023-10-17T03:06:52.372 DEBUG:teuthology.nuke:shortname: smithi169 2023-10-17T03:06:52.373 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-17T03:06:52.398 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi169.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rraja-2023-10-17_00:10:32-rbd:cli-debug-62994-distro-default-smithi/7430972', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-17 00:22:43.889635', 'locked_by': 'scheduled_rraja@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAaZyr1k69SkPlF/Xh4AZJUrxwBZlRxT2Tma5EXe7XCdHbtKN5MGae9jrpx30dbg96m2gGYzEUIuCckPOJylPVg='} 2023-10-17T03:06:52.455 INFO:teuthology.orchestra.console.smithi169:Power off 2023-10-17T03:06:52.456 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-17T03:06:52.478 DEBUG:teuthology.orchestra.console.smithi169:power off output: Chassis Power Control: Down/Off 2023-10-17T03:06:52.479 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-17T03:06:52.507 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2023-10-17T03:06:56.509 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-17T03:06:56.620 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2023-10-17T03:07:00.622 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-17T03:07:00.732 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is off 2023-10-17T03:07:00.833 INFO:teuthology.orchestra.console.smithi169:Power off completed 2023-10-17T03:07:00.939 INFO:teuthology.run:Summary data: description: rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/defaults msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} duration: 9277.166815519333 failure_reason: 'Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi169 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c3822f5f81e37dae2c0291e57dc72240858225fd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh''' flavor: default owner: scheduled_rraja@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=8ca8d2a4f81a409788e300315003a09a status: fail success: false 2023-10-17T03:07:00.940 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-17T03:07:01.036 INFO:teuthology.run:FAIL